{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# STGCN：路网交通预测\n",
    "## 论文资料\n",
    "- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic](https://arxiv.org/abs/1709.04875v4)\n",
    "- [原文代码](https://github.com/VeritasYin/STGCN_IJCAI-18)\n",
    "## 参考资料\n",
    "- [论文阅读笔记](https://davidham3.github.io/blog/2018/05/10/spatio-temporal-graph-convolutional-networks-a-deep-learning-framework-for-traffic/)\n",
    "- [mxnet复现代码](https://github.com/Davidham3/STGCN)\n",
    "- [STGCN论文详解](https://zhuanlan.zhihu.com/p/78259670)\n",
    "## 简介\n",
    "实时精确的交通预测对城市交通管控和引导很重要。由于交通流的强非线性以及复杂性，传统方法并不能满足中长期预测的要求，而且传统方法经常忽略对时空数据的依赖。在这篇论文中，作者提出了一个新的深度学习框架，时空图卷积(Spatio-Temporal Graph Convolutional Networks)，来解决交通领域的时间序列预测问题。\n",
    "\n",
    "在交通研究中，交通流的基本变量，也就是速度、流量和密度，通常作为监控当前交通状态以及未来预测的指示指标。根据预测的长度，交通预测大体分为两个尺度：短期(5~30min)，中和长期预测(超过30min)。大多数流行的统计方法(比如，线性回归)可以在短期预测上表现的很好。然而，由于交通流的不确定性和复杂性，这些方法在相对长期的预测上不是那么的有效。\n",
    "\n",
    "交通预测是一个典型的时间序列预测问题，也就是预测在给定前M个观测样本接下来H个时间戳后最可能的交通流指标（比如速度或交通流）：\n",
    "\n",
    "$\\hat{v}_{t+1}, ..., \\hat{v}_{t+H} = \\mathop{\\arg\\min}_{v_{t+1},...,v_{t+H}}logP(v_{t+1},...,v_{t+H}\\vert v_{t-M+1},...v_t)$\n",
    "\n",
    "这里$v_t \\in \\mathbb{R}^n$是$n$个路段在时间戳$t$观察到的一个向量，每个元素记录了一条路段的历史观测数据。\n",
    "\n",
    "作者在一个图上定义了一个交通网络，并专注于结构化的交通时间序列。观测到的样本v_t间不是相互独立的，而是在图中两两相互连接的。因此，数据点$v_t$可以被视为定义在权重为$w_{ij}$，如下图展示的无向图（或有向图）$\\mathcal{G}$上的一个信号。在第$t$个时间戳，在图$\\mathcal{G_t}=(\\mathcal{V_t}, \\mathcal{\\varepsilon}, W), \\mathcal{V_t}$是当顶点的有限集，对应在交通网络中n个监测站；$\\epsilon$是边集，表示观测站之间的连通性；$W \\in \\mathbb{R^{n \\times n}}$表示$\\mathcal{G_t}$的邻接矩阵。\n",
    "\n",
    "![file](https://pic1.zhimg.com/v2-5d5efa123e08b8e7cca00b327843aeb7_1440w.jpg)\n",
    "\n",
    "> **关于图学习的概念，可以参考[PGL：Paddle带你走进图学习](https://aistudio.baidu.com/aistudio/projectdetail/413386)系列课程。**\n",
    "\n",
    "> 关于交通网络的时空相关性，也可以参考其它论文给出的更详细示意，如：\n",
    ">\n",
    "> [论文：面向交通流量预测的多组件时空图卷积网络](http://www.jos.org.cn/html/2019/3/5697.htm)\n",
    ">\n",
    "> 交通流量预测是典型的时空数据预测问题, 不同类别的交通数据内嵌于连续空间, 并且随时间动态变化, 因此, 有效提取时空相关性对解决这类问题至关重要.下图所示为流量数据(也可以是车速、车道占用率等其他交通数据)的时空相关性示意图, 时间维包含3个时间片, 空间维的6个节点(A~F)表示公路的网状结构。在空间维上, 节点的交通状况之间会相互影响(绿色虚线); 时间维上, 某节点历史不同时刻流量会对该节点未来不同时刻流量产生影响(蓝色虚线); 同时, 节点历史不同时刻的流量值也会对其关联节点未来不同时刻的流量产生影响(红色虚线)。可见, 交通流量在时空维度都存在很强的相关性。\n",
    ">\n",
    "> ![file](http://www.jos.org.cn/html/2019/3/PIC/rjxb-30-3-759-1.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 网络结构\n",
    "STGCN有多个时空卷积块组成，每一个都是像一个“三明治”结构的组成，有两个门序列卷积层和一个空间图卷积层在中间。\n",
    "\n",
    "![file](https://pic3.zhimg.com/80/v2-030389b5592ad95cc19e3546ae70510e_720w.jpg)\n",
    "\n",
    "STGCN的架构有两个时空卷积块和一个全连接的在末尾的输出层组成。每个ST-Conv块包含了两个时间门卷积层，中间有一个空间图卷积层。每个块中都使用了残差连接和bottleneck策略。输入$v_{t-M+1},…v_t$被ST-Conv块均匀（uniformly）处理，来获取时空依赖关系。全部特征由一个输出层来整合，生成最后的预测$\\hat{v}$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## PGL实现\n",
    "### 安装工具库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'PGL'...\n",
      "remote: Enumerating objects: 1824, done.\u001b[K\n",
      "remote: Counting objects: 100% (1824/1824), done.\u001b[K\n",
      "remote: Compressing objects: 100% (1224/1224), done.\u001b[K\n",
      "remote: Total 1824 (delta 977), reused 1061 (delta 528), pack-reused 0\u001b[K\n",
      "Receiving objects: 100% (1824/1824), 16.62 MiB | 3.40 MiB/s, done.\n",
      "Resolving deltas: 100% (977/977), done.\n",
      "Checking connectivity... done.\n"
     ]
    }
   ],
   "source": [
    "!git clone https://gitee.com/paddlepaddle/PGL.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.mirrors.ustc.edu.cn/simple/\n",
      "Collecting pgl\n",
      "\u001b[?25l  Downloading https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/3f/d9/3a9db4a342545b1270cedf0ef68685108b1cf8cd2143a6aa5ee13ec2febf/pgl-1.1.0-cp37-cp37m-manylinux1_x86_64.whl (7.9MB)\n",
      "\u001b[K     |████████████████████████████████| 7.9MB 48kB/s eta 0:00:013\n",
      "\u001b[?25hCollecting redis-py-cluster (from pgl)\n",
      "  Downloading https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/35/cb/29d44f7735af4fe9251afb6b5b173ec79c6e8f49cb6a61603e77a54ba658/redis_py_cluster-2.0.0-py2.py3-none-any.whl\n",
      "Requirement already satisfied: cython>=0.25.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pgl) (0.29)\n",
      "Requirement already satisfied: numpy>=1.16.4 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pgl) (1.16.4)\n",
      "Collecting redis<3.1.0,>=3.0.0 (from redis-py-cluster->pgl)\n",
      "\u001b[?25l  Downloading https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f5/00/5253aff5e747faf10d8ceb35fb5569b848cde2fdc13685d42fcf63118bbc/redis-3.0.1-py2.py3-none-any.whl (61kB)\n",
      "\u001b[K     |████████████████████████████████| 71kB 32.9MB/s eta 0:00:01\n",
      "\u001b[?25hInstalling collected packages: redis, redis-py-cluster, pgl\n",
      "Successfully installed pgl-1.1.0 redis-3.0.1 redis-py-cluster-2.0.0\n"
     ]
    }
   ],
   "source": [
    "!pip install pgl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 数据集准备\n",
    "PeMSD7是Caltrans Performance Measurement System(PeMS)通过超过39000个监测站实时获取的数据，这些监测站分布在加州高速公路系统主要的都市部分。数据是30秒的数据样本聚合成5分钟一次的数据。作者在加州的District 7随机选取了一个小的和一个大的范围作为数据源，分别有228和1026个监测站，分别命名为PeMSD7(S)和PeMSD7(L)。PeMSD7数据集的时间范围是2012年五月和六月的周末，选取了第一个月的车速速度记录作为训练集，剩下的分别做验证和测试。\n",
    "\n",
    "![file](https://ai-studio-static-online.cdn.bcebos.com/0ad23394ba10462993573f0c0e26014c80cdec467ded4176a4683e895bb54ac9)\n",
    "\n",
    "\n",
    "### 数据预处理\n",
    "路网中的每个顶点（传感器）每天就有288个数据点。数据清理后使用了线性插值的方法来填补缺失值。通过核对相关性，每条路的方向和OD(origin-destination)点，环路系统可以被数值化成一个有向图。\n",
    "\n",
    "在PeMSD7，路网的邻接矩阵通过交通网络中的监测站的距离来计算。带权邻接矩阵W通过以下公式计算：\n",
    "\n",
    "$w_{ij} = \\begin{cases}\n",
    "\\exp{(-\\frac{d^2_{ij}}{\\sigma^2})}&,i \\neq j \\ \\rm and \\exp{(-\\frac{d^2_{ij}}{\\sigma^2}) \\geq \\epsilon} \\\\\n",
    "0&, \\rm otherwise\n",
    "\\end{cases}$\n",
    "\n",
    "其中$w_{ij}$是边的权重，通过$d_{ij}$得到，也就是$i$和$j$之间的距离。$\\sigma^2$和$\\epsilon$是来控制矩阵W的分布和稀疏性的阈值，文中用了10和0.5。$W$的可视化在上图的右侧。\n",
    "\n",
    "[作者代码](https://github.com/VeritasYin/STGCN_IJCAI-18)中给出了PeMS-M数据集的压缩包，本文已将数据集解压放在PeMS-M目录下。\n",
    "```\n",
    "PeMS-M/\n",
    "    -- W_228.csv\n",
    "    -- V_228.csv\n",
    "```\n",
    "其中，`V_228.csv`是传感器记录的车速信息，`W_228.csv`是已经处理好的邻接矩阵。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 开始训练\n",
    "> 由于没能跑通PGL给出的示例代码，这里对`PGL/examples/stgcn`目录下的`data_loader/data_utils.py`和`main.py`稍作修改，参考[mxnet复现代码](https://github.com/Davidham3/STGCN)重写了`data_loader/data_utils.py`里的部分函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/aistudio/PGL/examples/stgcn\n"
     ]
    }
   ],
   "source": [
    "%cd PGL/examples/stgcn/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] 2020-06-27 10:25:01,538 [     main.py:  174]:\tNamespace(Ks=3, Kt=3, adj_mat_file='/home/aistudio/PeMS-M/W_228.csv', batch_size=10, blocks=[[1, 32, 64], [64, 32, 128]], epochs=5, inf_mode='sep', input_file='/home/aistudio/PeMS-M/V_228.csv', keep_prob=1.0, lr=0.001, n_his=9, n_pred=3, n_route=228, opt='ADAM', output_path='./outputs/', save=1, use_cuda=True)\n",
      "[INFO] 2020-06-27 10:25:04,711 [     main.py:   41]:\t{'mean': 59.49499979949002, 'std': 13.170890048189376}\n",
      "[INFO] 2020-06-27 10:25:04,712 [     main.py:   42]:\t7583\n",
      "[INFO] 2020-06-27 10:25:08,762 [     main.py:  109]:\tepoch 1 | step 0 | lr 0.001000 | loss 37735.941406\n",
      "[INFO] 2020-06-27 10:25:13,379 [     main.py:  109]:\tepoch 1 | step 5 | lr 0.001000 | loss 18736.484375\n",
      "[INFO] 2020-06-27 10:25:17,919 [     main.py:  109]:\tepoch 1 | step 10 | lr 0.001000 | loss 18630.222656\n",
      "[INFO] 2020-06-27 10:25:22,470 [     main.py:  109]:\tepoch 1 | step 15 | lr 0.001000 | loss 13020.451172\n",
      "[INFO] 2020-06-27 10:25:27,010 [     main.py:  109]:\tepoch 1 | step 20 | lr 0.001000 | loss 18053.085938\n",
      "[INFO] 2020-06-27 10:25:31,652 [     main.py:  109]:\tepoch 1 | step 25 | lr 0.001000 | loss 11902.833008\n",
      "[INFO] 2020-06-27 10:25:36,186 [     main.py:  109]:\tepoch 1 | step 30 | lr 0.001000 | loss 11125.682617\n",
      "[INFO] 2020-06-27 10:25:40,800 [     main.py:  109]:\tepoch 1 | step 35 | lr 0.001000 | loss 9599.474609\n",
      "[INFO] 2020-06-27 10:25:45,343 [     main.py:  109]:\tepoch 1 | step 40 | lr 0.001000 | loss 11258.325195\n",
      "[INFO] 2020-06-27 10:25:49,907 [     main.py:  109]:\tepoch 1 | step 45 | lr 0.001000 | loss 6970.545898\n",
      "[INFO] 2020-06-27 10:25:54,454 [     main.py:  109]:\tepoch 1 | step 50 | lr 0.001000 | loss 7194.029785\n",
      "[INFO] 2020-06-27 10:25:58,999 [     main.py:  109]:\tepoch 1 | step 55 | lr 0.001000 | loss 9328.666016\n",
      "[INFO] 2020-06-27 10:26:03,594 [     main.py:  109]:\tepoch 1 | step 60 | lr 0.001000 | loss 5624.453613\n",
      "[INFO] 2020-06-27 10:26:08,158 [     main.py:  109]:\tepoch 1 | step 65 | lr 0.001000 | loss 15484.669922\n",
      "[INFO] 2020-06-27 10:26:12,727 [     main.py:  109]:\tepoch 1 | step 70 | lr 0.001000 | loss 9598.880859\n",
      "[INFO] 2020-06-27 10:26:17,282 [     main.py:  109]:\tepoch 1 | step 75 | lr 0.001000 | loss 14367.545898\n",
      "[INFO] 2020-06-27 10:26:21,838 [     main.py:  109]:\tepoch 1 | step 80 | lr 0.001000 | loss 9098.638672\n",
      "[INFO] 2020-06-27 10:26:26,381 [     main.py:  109]:\tepoch 1 | step 85 | lr 0.001000 | loss 11955.281250\n",
      "[INFO] 2020-06-27 10:26:30,940 [     main.py:  109]:\tepoch 1 | step 90 | lr 0.001000 | loss 7362.460938\n",
      "[INFO] 2020-06-27 10:26:35,509 [     main.py:  109]:\tepoch 1 | step 95 | lr 0.001000 | loss 5939.260254\n",
      "[INFO] 2020-06-27 10:26:40,064 [     main.py:  109]:\tepoch 1 | step 100 | lr 0.001000 | loss 16125.000977\n",
      "[INFO] 2020-06-27 10:26:44,755 [     main.py:  109]:\tepoch 1 | step 105 | lr 0.001000 | loss 10326.945312\n",
      "[INFO] 2020-06-27 10:26:49,317 [     main.py:  109]:\tepoch 1 | step 110 | lr 0.001000 | loss 16809.191406\n",
      "[INFO] 2020-06-27 10:26:53,874 [     main.py:  109]:\tepoch 1 | step 115 | lr 0.001000 | loss 8291.509766\n",
      "[INFO] 2020-06-27 10:26:58,435 [     main.py:  109]:\tepoch 1 | step 120 | lr 0.001000 | loss 8192.952148\n",
      "[INFO] 2020-06-27 10:27:03,006 [     main.py:  109]:\tepoch 1 | step 125 | lr 0.001000 | loss 16092.328125\n",
      "[INFO] 2020-06-27 10:27:07,579 [     main.py:  109]:\tepoch 1 | step 130 | lr 0.001000 | loss 6669.453125\n",
      "[INFO] 2020-06-27 10:27:12,164 [     main.py:  109]:\tepoch 1 | step 135 | lr 0.001000 | loss 12109.406250\n",
      "[INFO] 2020-06-27 10:27:16,739 [     main.py:  109]:\tepoch 1 | step 140 | lr 0.001000 | loss 6615.083008\n",
      "[INFO] 2020-06-27 10:27:21,313 [     main.py:  109]:\tepoch 1 | step 145 | lr 0.001000 | loss 10833.454102\n",
      "[INFO] 2020-06-27 10:27:25,886 [     main.py:  109]:\tepoch 1 | step 150 | lr 0.001000 | loss 9129.137695\n",
      "[INFO] 2020-06-27 10:27:30,512 [     main.py:  109]:\tepoch 1 | step 155 | lr 0.001000 | loss 10883.227539\n",
      "[INFO] 2020-06-27 10:27:35,085 [     main.py:  109]:\tepoch 1 | step 160 | lr 0.001000 | loss 6452.153320\n",
      "[INFO] 2020-06-27 10:27:39,677 [     main.py:  109]:\tepoch 1 | step 165 | lr 0.001000 | loss 7721.391113\n",
      "[INFO] 2020-06-27 10:27:44,251 [     main.py:  109]:\tepoch 1 | step 170 | lr 0.001000 | loss 6851.547852\n",
      "[INFO] 2020-06-27 10:27:48,833 [     main.py:  109]:\tepoch 1 | step 175 | lr 0.001000 | loss 6517.803223\n",
      "[INFO] 2020-06-27 10:27:53,413 [     main.py:  109]:\tepoch 1 | step 180 | lr 0.001000 | loss 5980.505859\n",
      "[INFO] 2020-06-27 10:27:57,990 [     main.py:  109]:\tepoch 1 | step 185 | lr 0.001000 | loss 9012.660156\n",
      "[INFO] 2020-06-27 10:28:02,709 [     main.py:  109]:\tepoch 1 | step 190 | lr 0.001000 | loss 8939.193359\n",
      "[INFO] 2020-06-27 10:28:07,284 [     main.py:  109]:\tepoch 1 | step 195 | lr 0.001000 | loss 6330.277832\n",
      "[INFO] 2020-06-27 10:28:11,885 [     main.py:  109]:\tepoch 1 | step 200 | lr 0.001000 | loss 6860.510742\n",
      "[INFO] 2020-06-27 10:28:16,480 [     main.py:  109]:\tepoch 1 | step 205 | lr 0.001000 | loss 7429.390137\n",
      "[INFO] 2020-06-27 10:28:21,176 [     main.py:  109]:\tepoch 1 | step 210 | lr 0.001000 | loss 5395.461426\n",
      "[INFO] 2020-06-27 10:28:25,846 [     main.py:  109]:\tepoch 1 | step 215 | lr 0.001000 | loss 7995.499023\n",
      "[INFO] 2020-06-27 10:28:30,504 [     main.py:  109]:\tepoch 1 | step 220 | lr 0.001000 | loss 10387.447266\n",
      "[INFO] 2020-06-27 10:28:35,176 [     main.py:  109]:\tepoch 1 | step 225 | lr 0.001000 | loss 13723.822266\n",
      "[INFO] 2020-06-27 10:28:39,844 [     main.py:  109]:\tepoch 1 | step 230 | lr 0.001000 | loss 6553.451172\n",
      "[INFO] 2020-06-27 10:28:44,502 [     main.py:  109]:\tepoch 1 | step 235 | lr 0.001000 | loss 5084.278320\n",
      "[INFO] 2020-06-27 10:28:49,158 [     main.py:  109]:\tepoch 1 | step 240 | lr 0.001000 | loss 11612.527344\n",
      "[INFO] 2020-06-27 10:28:53,817 [     main.py:  109]:\tepoch 1 | step 245 | lr 0.001000 | loss 9042.700195\n",
      "[INFO] 2020-06-27 10:28:58,483 [     main.py:  109]:\tepoch 1 | step 250 | lr 0.001000 | loss 5726.862793\n",
      "[INFO] 2020-06-27 10:29:03,150 [     main.py:  109]:\tepoch 1 | step 255 | lr 0.001000 | loss 8533.449219\n",
      "[INFO] 2020-06-27 10:29:07,811 [     main.py:  109]:\tepoch 1 | step 260 | lr 0.001000 | loss 6600.126953\n",
      "[INFO] 2020-06-27 10:29:12,647 [     main.py:  109]:\tepoch 1 | step 265 | lr 0.001000 | loss 10771.875977\n",
      "[INFO] 2020-06-27 10:29:17,364 [     main.py:  109]:\tepoch 1 | step 270 | lr 0.001000 | loss 7744.235840\n",
      "[INFO] 2020-06-27 10:29:22,020 [     main.py:  109]:\tepoch 1 | step 275 | lr 0.001000 | loss 10922.890625\n",
      "[INFO] 2020-06-27 10:29:26,669 [     main.py:  109]:\tepoch 1 | step 280 | lr 0.001000 | loss 10097.871094\n",
      "[INFO] 2020-06-27 10:29:31,338 [     main.py:  109]:\tepoch 1 | step 285 | lr 0.001000 | loss 4943.081055\n",
      "[INFO] 2020-06-27 10:29:35,995 [     main.py:  109]:\tepoch 1 | step 290 | lr 0.001000 | loss 5377.001953\n",
      "[INFO] 2020-06-27 10:29:40,655 [     main.py:  109]:\tepoch 1 | step 295 | lr 0.001000 | loss 7947.578613\n",
      "[INFO] 2020-06-27 10:29:45,320 [     main.py:  109]:\tepoch 1 | step 300 | lr 0.001000 | loss 10941.259766\n",
      "[INFO] 2020-06-27 10:29:49,972 [     main.py:  109]:\tepoch 1 | step 305 | lr 0.001000 | loss 4228.699707\n",
      "[INFO] 2020-06-27 10:29:54,641 [     main.py:  109]:\tepoch 1 | step 310 | lr 0.001000 | loss 6041.813965\n",
      "[INFO] 2020-06-27 10:29:59,298 [     main.py:  109]:\tepoch 1 | step 315 | lr 0.001000 | loss 7565.720703\n",
      "[INFO] 2020-06-27 10:30:03,955 [     main.py:  109]:\tepoch 1 | step 320 | lr 0.001000 | loss 3052.132812\n",
      "[INFO] 2020-06-27 10:30:08,664 [     main.py:  109]:\tepoch 1 | step 325 | lr 0.001000 | loss 7258.500000\n",
      "[INFO] 2020-06-27 10:30:13,318 [     main.py:  109]:\tepoch 1 | step 330 | lr 0.001000 | loss 7258.628906\n",
      "[INFO] 2020-06-27 10:30:17,971 [     main.py:  109]:\tepoch 1 | step 335 | lr 0.001000 | loss 10175.652344\n",
      "[INFO] 2020-06-27 10:30:22,633 [     main.py:  109]:\tepoch 1 | step 340 | lr 0.001000 | loss 6401.521973\n",
      "[INFO] 2020-06-27 10:30:27,445 [     main.py:  109]:\tepoch 1 | step 345 | lr 0.001000 | loss 3520.176270\n",
      "[INFO] 2020-06-27 10:30:32,091 [     main.py:  109]:\tepoch 1 | step 350 | lr 0.001000 | loss 5676.759766\n",
      "[INFO] 2020-06-27 10:30:36,752 [     main.py:  109]:\tepoch 1 | step 355 | lr 0.001000 | loss 7543.050781\n",
      "[INFO] 2020-06-27 10:30:41,407 [     main.py:  109]:\tepoch 1 | step 360 | lr 0.001000 | loss 12806.579102\n",
      "[INFO] 2020-06-27 10:30:46,057 [     main.py:  109]:\tepoch 1 | step 365 | lr 0.001000 | loss 4636.592285\n",
      "[INFO] 2020-06-27 10:30:50,719 [     main.py:  109]:\tepoch 1 | step 370 | lr 0.001000 | loss 6429.077637\n",
      "[INFO] 2020-06-27 10:30:55,372 [     main.py:  109]:\tepoch 1 | step 375 | lr 0.001000 | loss 6336.032715\n",
      "[INFO] 2020-06-27 10:31:00,031 [     main.py:  109]:\tepoch 1 | step 380 | lr 0.001000 | loss 8332.381836\n",
      "[INFO] 2020-06-27 10:31:04,703 [     main.py:  109]:\tepoch 1 | step 385 | lr 0.001000 | loss 6266.290039\n",
      "[INFO] 2020-06-27 10:31:09,363 [     main.py:  109]:\tepoch 1 | step 390 | lr 0.001000 | loss 6936.368164\n",
      "[INFO] 2020-06-27 10:31:14,036 [     main.py:  109]:\tepoch 1 | step 395 | lr 0.001000 | loss 4875.728027\n",
      "[INFO] 2020-06-27 10:31:18,712 [     main.py:  109]:\tepoch 1 | step 400 | lr 0.001000 | loss 9043.921875\n",
      "[INFO] 2020-06-27 10:31:23,378 [     main.py:  109]:\tepoch 1 | step 405 | lr 0.001000 | loss 6416.957520\n",
      "[INFO] 2020-06-27 10:31:28,052 [     main.py:  109]:\tepoch 1 | step 410 | lr 0.001000 | loss 6685.189453\n",
      "[INFO] 2020-06-27 10:31:32,723 [     main.py:  109]:\tepoch 1 | step 415 | lr 0.001000 | loss 8260.904297\n",
      "[INFO] 2020-06-27 10:31:37,545 [     main.py:  109]:\tepoch 1 | step 420 | lr 0.001000 | loss 5355.673828\n",
      "[INFO] 2020-06-27 10:31:42,203 [     main.py:  109]:\tepoch 1 | step 425 | lr 0.001000 | loss 3357.874023\n",
      "[INFO] 2020-06-27 10:31:46,868 [     main.py:  109]:\tepoch 1 | step 430 | lr 0.001000 | loss 6597.340820\n",
      "[INFO] 2020-06-27 10:31:51,537 [     main.py:  109]:\tepoch 1 | step 435 | lr 0.001000 | loss 5140.948242\n",
      "[INFO] 2020-06-27 10:31:56,203 [     main.py:  109]:\tepoch 1 | step 440 | lr 0.001000 | loss 14872.176758\n",
      "[INFO] 2020-06-27 10:32:00,870 [     main.py:  109]:\tepoch 1 | step 445 | lr 0.001000 | loss 5137.071777\n",
      "[INFO] 2020-06-27 10:32:05,536 [     main.py:  109]:\tepoch 1 | step 450 | lr 0.001000 | loss 3784.291504\n",
      "[INFO] 2020-06-27 10:32:10,207 [     main.py:  109]:\tepoch 1 | step 455 | lr 0.001000 | loss 5173.709473\n",
      "[INFO] 2020-06-27 10:32:14,881 [     main.py:  109]:\tepoch 1 | step 460 | lr 0.001000 | loss 5030.171387\n",
      "[INFO] 2020-06-27 10:32:19,553 [     main.py:  109]:\tepoch 1 | step 465 | lr 0.001000 | loss 5143.199707\n",
      "[INFO] 2020-06-27 10:32:24,226 [     main.py:  109]:\tepoch 1 | step 470 | lr 0.001000 | loss 9018.249023\n",
      "[INFO] 2020-06-27 10:32:28,893 [     main.py:  109]:\tepoch 1 | step 475 | lr 0.001000 | loss 4279.703613\n",
      "[INFO] 2020-06-27 10:32:33,558 [     main.py:  109]:\tepoch 1 | step 480 | lr 0.001000 | loss 7475.649902\n",
      "[INFO] 2020-06-27 10:32:38,226 [     main.py:  109]:\tepoch 1 | step 485 | lr 0.001000 | loss 8123.766113\n",
      "[INFO] 2020-06-27 10:32:42,905 [     main.py:  109]:\tepoch 1 | step 490 | lr 0.001000 | loss 7915.218750\n",
      "[INFO] 2020-06-27 10:32:47,742 [     main.py:  109]:\tepoch 1 | step 495 | lr 0.001000 | loss 5127.339844\n",
      "[INFO] 2020-06-27 10:32:52,519 [     main.py:  109]:\tepoch 1 | step 500 | lr 0.001000 | loss 4054.055908\n",
      "[INFO] 2020-06-27 10:32:57,171 [     main.py:  109]:\tepoch 1 | step 505 | lr 0.001000 | loss 4345.636719\n",
      "[INFO] 2020-06-27 10:33:01,835 [     main.py:  109]:\tepoch 1 | step 510 | lr 0.001000 | loss 7016.473633\n",
      "[INFO] 2020-06-27 10:33:06,489 [     main.py:  109]:\tepoch 1 | step 515 | lr 0.001000 | loss 7494.670410\n",
      "[INFO] 2020-06-27 10:33:11,164 [     main.py:  109]:\tepoch 1 | step 520 | lr 0.001000 | loss 6326.111328\n",
      "[INFO] 2020-06-27 10:33:15,824 [     main.py:  109]:\tepoch 1 | step 525 | lr 0.001000 | loss 5116.449219\n",
      "[INFO] 2020-06-27 10:33:20,487 [     main.py:  109]:\tepoch 1 | step 530 | lr 0.001000 | loss 5296.669922\n",
      "[INFO] 2020-06-27 10:33:25,145 [     main.py:  109]:\tepoch 1 | step 535 | lr 0.001000 | loss 6458.634766\n",
      "[INFO] 2020-06-27 10:33:29,802 [     main.py:  109]:\tepoch 1 | step 540 | lr 0.001000 | loss 5442.626953\n",
      "[INFO] 2020-06-27 10:33:34,467 [     main.py:  109]:\tepoch 1 | step 545 | lr 0.001000 | loss 6636.049316\n",
      "[INFO] 2020-06-27 10:33:39,140 [     main.py:  109]:\tepoch 1 | step 550 | lr 0.001000 | loss 8034.599121\n",
      "[INFO] 2020-06-27 10:33:43,804 [     main.py:  109]:\tepoch 1 | step 555 | lr 0.001000 | loss 5106.713867\n",
      "[INFO] 2020-06-27 10:33:48,467 [     main.py:  109]:\tepoch 1 | step 560 | lr 0.001000 | loss 7601.837891\n",
      "[INFO] 2020-06-27 10:33:53,135 [     main.py:  109]:\tepoch 1 | step 565 | lr 0.001000 | loss 5841.292969\n",
      "[INFO] 2020-06-27 10:33:57,791 [     main.py:  109]:\tepoch 1 | step 570 | lr 0.001000 | loss 2632.594727\n",
      "[INFO] 2020-06-27 10:34:02,614 [     main.py:  109]:\tepoch 1 | step 575 | lr 0.001000 | loss 5246.858398\n",
      "[INFO] 2020-06-27 10:34:07,272 [     main.py:  109]:\tepoch 1 | step 580 | lr 0.001000 | loss 6611.925293\n",
      "[INFO] 2020-06-27 10:34:11,931 [     main.py:  109]:\tepoch 1 | step 585 | lr 0.001000 | loss 6576.196289\n",
      "[INFO] 2020-06-27 10:34:16,589 [     main.py:  109]:\tepoch 1 | step 590 | lr 0.001000 | loss 7234.600586\n",
      "[INFO] 2020-06-27 10:34:21,246 [     main.py:  109]:\tepoch 1 | step 595 | lr 0.001000 | loss 7896.226074\n",
      "[INFO] 2020-06-27 10:34:25,895 [     main.py:  109]:\tepoch 1 | step 600 | lr 0.001000 | loss 7527.335449\n",
      "[INFO] 2020-06-27 10:34:30,553 [     main.py:  109]:\tepoch 1 | step 605 | lr 0.001000 | loss 3311.047363\n",
      "[INFO] 2020-06-27 10:34:35,223 [     main.py:  109]:\tepoch 1 | step 610 | lr 0.001000 | loss 3356.222168\n",
      "[INFO] 2020-06-27 10:34:39,879 [     main.py:  109]:\tepoch 1 | step 615 | lr 0.001000 | loss 6924.463379\n",
      "[INFO] 2020-06-27 10:34:44,537 [     main.py:  109]:\tepoch 1 | step 620 | lr 0.001000 | loss 4124.032715\n",
      "[INFO] 2020-06-27 10:34:49,202 [     main.py:  109]:\tepoch 1 | step 625 | lr 0.001000 | loss 5382.447754\n",
      "[INFO] 2020-06-27 10:34:53,861 [     main.py:  109]:\tepoch 1 | step 630 | lr 0.001000 | loss 5890.448242\n",
      "[INFO] 2020-06-27 10:34:58,557 [     main.py:  109]:\tepoch 1 | step 635 | lr 0.001000 | loss 4065.146484\n",
      "[INFO] 2020-06-27 10:35:03,221 [     main.py:  109]:\tepoch 1 | step 640 | lr 0.001000 | loss 4500.604980\n",
      "[INFO] 2020-06-27 10:35:07,893 [     main.py:  109]:\tepoch 1 | step 645 | lr 0.001000 | loss 4620.673828\n",
      "[INFO] 2020-06-27 10:35:12,724 [     main.py:  109]:\tepoch 1 | step 650 | lr 0.001000 | loss 4947.170898\n",
      "[INFO] 2020-06-27 10:35:17,379 [     main.py:  109]:\tepoch 1 | step 655 | lr 0.001000 | loss 6553.805664\n",
      "[INFO] 2020-06-27 10:35:22,033 [     main.py:  109]:\tepoch 1 | step 660 | lr 0.001000 | loss 6870.678223\n",
      "[INFO] 2020-06-27 10:35:26,678 [     main.py:  109]:\tepoch 1 | step 665 | lr 0.001000 | loss 6609.209961\n",
      "[INFO] 2020-06-27 10:35:31,345 [     main.py:  109]:\tepoch 1 | step 670 | lr 0.001000 | loss 5503.839355\n",
      "[INFO] 2020-06-27 10:35:36,000 [     main.py:  109]:\tepoch 1 | step 675 | lr 0.001000 | loss 4718.570312\n",
      "[INFO] 2020-06-27 10:35:40,654 [     main.py:  109]:\tepoch 1 | step 680 | lr 0.001000 | loss 5936.292480\n",
      "[INFO] 2020-06-27 10:35:45,322 [     main.py:  109]:\tepoch 1 | step 685 | lr 0.001000 | loss 4619.918945\n",
      "[INFO] 2020-06-27 10:35:49,972 [     main.py:  109]:\tepoch 1 | step 690 | lr 0.001000 | loss 5239.690430\n",
      "[INFO] 2020-06-27 10:35:54,642 [     main.py:  109]:\tepoch 1 | step 695 | lr 0.001000 | loss 7379.635742\n",
      "[INFO] 2020-06-27 10:35:59,294 [     main.py:  109]:\tepoch 1 | step 700 | lr 0.001000 | loss 3559.312500\n",
      "[INFO] 2020-06-27 10:36:03,952 [     main.py:  109]:\tepoch 1 | step 705 | lr 0.001000 | loss 4938.158691\n",
      "[INFO] 2020-06-27 10:36:08,614 [     main.py:  109]:\tepoch 1 | step 710 | lr 0.001000 | loss 6022.327637\n",
      "[INFO] 2020-06-27 10:36:13,333 [     main.py:  109]:\tepoch 1 | step 715 | lr 0.001000 | loss 3227.090088\n",
      "[INFO] 2020-06-27 10:36:17,996 [     main.py:  109]:\tepoch 1 | step 720 | lr 0.001000 | loss 4335.575195\n",
      "[INFO] 2020-06-27 10:36:22,672 [     main.py:  109]:\tepoch 1 | step 725 | lr 0.001000 | loss 7386.066895\n",
      "[INFO] 2020-06-27 10:36:27,489 [     main.py:  109]:\tepoch 1 | step 730 | lr 0.001000 | loss 4961.151367\n",
      "[INFO] 2020-06-27 10:36:32,147 [     main.py:  109]:\tepoch 1 | step 735 | lr 0.001000 | loss 3725.366699\n",
      "[INFO] 2020-06-27 10:36:36,805 [     main.py:  109]:\tepoch 1 | step 740 | lr 0.001000 | loss 5296.148926\n",
      "[INFO] 2020-06-27 10:36:41,466 [     main.py:  109]:\tepoch 1 | step 745 | lr 0.001000 | loss 3624.866943\n",
      "[INFO] 2020-06-27 10:36:46,128 [     main.py:  109]:\tepoch 1 | step 750 | lr 0.001000 | loss 3798.115234\n",
      "[INFO] 2020-06-27 10:36:50,798 [     main.py:  109]:\tepoch 1 | step 755 | lr 0.001000 | loss 4630.307129\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time Step 3: MAPE 13.896%, 12.535%; MAE  4.914, 4.759; RMSE  7.973,  7.636.\n",
      "Time Step 1: MAPE 11.567%; MAE  4.552; RMSE  7.401.\n",
      "Time Step 2: MAPE 11.927%; MAE  4.595; RMSE  7.437.\n",
      "Time Step 3: MAPE 12.535%; MAE  4.759; RMSE  7.636.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] 2020-06-27 10:44:26,944 [     main.py:  109]:\tepoch 2 | step 0 | lr 0.001000 | loss 5626.972656\n",
      "[INFO] 2020-06-27 10:44:31,626 [     main.py:  109]:\tepoch 2 | step 5 | lr 0.001000 | loss 5328.616699\n",
      "[INFO] 2020-06-27 10:44:36,273 [     main.py:  109]:\tepoch 2 | step 10 | lr 0.001000 | loss 4990.459961\n",
      "[INFO] 2020-06-27 10:44:40,917 [     main.py:  109]:\tepoch 2 | step 15 | lr 0.001000 | loss 4208.468750\n",
      "[INFO] 2020-06-27 10:44:45,592 [     main.py:  109]:\tepoch 2 | step 20 | lr 0.001000 | loss 5790.794922\n",
      "[INFO] 2020-06-27 10:44:50,268 [     main.py:  109]:\tepoch 2 | step 25 | lr 0.001000 | loss 4302.013672\n",
      "[INFO] 2020-06-27 10:44:54,961 [     main.py:  109]:\tepoch 2 | step 30 | lr 0.001000 | loss 6110.498535\n",
      "[INFO] 2020-06-27 10:44:59,642 [     main.py:  109]:\tepoch 2 | step 35 | lr 0.001000 | loss 6567.183594\n",
      "[INFO] 2020-06-27 10:45:04,327 [     main.py:  109]:\tepoch 2 | step 40 | lr 0.001000 | loss 3542.793457\n",
      "[INFO] 2020-06-27 10:45:08,983 [     main.py:  109]:\tepoch 2 | step 45 | lr 0.001000 | loss 4186.571777\n"
     ]
    }
   ],
   "source": [
    "%run main.py --use_cuda --input_file /home/aistudio/PeMS-M/V_228.csv --adj_mat_file /home/aistudio/PeMS-M/W_228.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# STGCN：高致病性传染病传播趋势预测基线系统学习\n",
    "- [比赛链接](https://aistudio.baidu.com/aistudio/competition/detail/36)\n",
    "- [基于飞桨PGL的基线系统](https://aistudio.baidu.com/aistudio/projectdetail/464528)\n",
    "\n",
    "在该场景中，官方baseline使用STGCN进行传播趋势预测，并没有改动图结构和STGCN网络，重点更多在数据预处理上。与交通流量的自回归不同的是，增加了label也就是感染人数。\n",
    "\n",
    "本文基于官方baseline略作修改，并增加了数据预处理阶段的注释，目录结构如下：\n",
    "```\n",
    "work/\n",
    "  -- dataset/\n",
    "  -- dataloader.py \n",
    "  -- Dataset.py\n",
    "  -- graph.py\n",
    "  -- model.py\n",
    "  -- main.py\n",
    "```\n",
    "\n",
    "## 解压数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/aistudio/work\n"
     ]
    }
   ],
   "source": [
    "%cd /home/aistudio/work"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  /home/aistudio/data/data33637/train_data.zip\n",
      "   creating: ./dataset/train_data/\n",
      "   creating: ./dataset/train_data/city_C/\n",
      "  inflating: ./dataset/train_data/city_C/migration.csv  \n",
      "  inflating: ./dataset/train_data/city_C/density.csv  \n",
      "  inflating: ./dataset/train_data/city_C/transfer.csv  \n",
      "  inflating: ./dataset/train_data/city_C/weather.csv  \n",
      "  inflating: ./dataset/train_data/city_C/grid_attr.csv  \n",
      "  inflating: ./dataset/train_data/city_C/infection.csv  \n",
      "   creating: ./dataset/train_data/city_D/\n",
      "  inflating: ./dataset/train_data/city_D/density.csv  \n",
      "  inflating: ./dataset/train_data/city_D/migration.csv  \n",
      "  inflating: ./dataset/train_data/city_D/transfer.csv  \n",
      "  inflating: ./dataset/train_data/city_D/weather.csv  \n",
      "  inflating: ./dataset/train_data/city_D/grid_attr.csv  \n",
      "  inflating: ./dataset/train_data/city_D/infection.csv  \n",
      "   creating: ./dataset/train_data/city_E/\n",
      "  inflating: ./dataset/train_data/city_E/density.csv  \n",
      "  inflating: ./dataset/train_data/city_E/migration.csv  \n",
      "  inflating: ./dataset/train_data/city_E/transfer.csv  \n",
      "  inflating: ./dataset/train_data/city_E/weather.csv  \n",
      "  inflating: ./dataset/train_data/city_E/grid_attr.csv  \n",
      "  inflating: ./dataset/train_data/city_E/infection.csv  \n",
      "  inflating: ./dataset/train_data/submission.csv  \n",
      "   creating: ./dataset/train_data/city_A/\n",
      "  inflating: ./dataset/train_data/city_A/density.csv  \n",
      "  inflating: ./dataset/train_data/city_A/migration.csv  \n",
      "  inflating: ./dataset/train_data/city_A/transfer.csv  \n",
      "  inflating: ./dataset/train_data/city_A/weather.csv  \n",
      "  inflating: ./dataset/train_data/city_A/grid_attr.csv  \n",
      "  inflating: ./dataset/train_data/city_A/infection.csv  \n",
      "   creating: ./dataset/train_data/city_B/\n",
      "  inflating: ./dataset/train_data/city_B/density.csv  \n",
      "  inflating: ./dataset/train_data/city_B/migration.csv  \n",
      "  inflating: ./dataset/train_data/city_B/transfer.csv  \n",
      "  inflating: ./dataset/train_data/city_B/weather.csv  \n",
      "  inflating: ./dataset/train_data/city_B/grid_attr.csv  \n",
      "  inflating: ./dataset/train_data/city_B/infection.csv  \n"
     ]
    }
   ],
   "source": [
    "# unzip dataset\n",
    "%mkdir ./dataset/\n",
    "!unzip /home/aistudio/data/data33637/train_data.zip -d ./dataset/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 引入工具库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"\"\"data processing\n",
    "\"\"\"\n",
    "import os\n",
    "import sys\n",
    "import argparse\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from collections import defaultdict, OrderedDict\n",
    "import pdb\n",
    "import gc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 获取指定城市每个地点经纬度以及归属区域\n",
    "根据`grid_attr.csv`给出的城市地点经纬度详细信息，将其转为`<key,value>`的形式。\n",
    "\n",
    "其实这种做法有个前提，那就是每个城市的`grid_attr.csv`包括了迁移数据中起始点和到达点的全部经纬度。假设在真实场景下，可以考虑用电子围栏。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def get_grid_dict(city_path, city_name):\n",
    "    d = {}\n",
    "    with open(os.path.join(city_path, 'grid_attr.csv'), 'r') as f:\n",
    "        for line in f:\n",
    "            items = line.strip().split(',')\n",
    "            axis = \",\".join(items[0:2])\n",
    "            ID = items[2]\n",
    "            d[axis] = \"_\".join([city_name, ID])\n",
    "    # print(d)\n",
    "    # d = {'x,y': ID}\n",
    "    return d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 计算市内区域迁移指数\n",
    "反映人群迁移情况的`transfer.csv`里起始点和到达点都是经纬度，这里需要将其换算为所属的城市区域ID，通过查找`grid_dict`里面的`<key,value>`列表实现。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def coord2ID(data_path, city_name, output_path):\n",
    "    city_path = os.path.join(data_path, \"city_%s\" % city_name)\n",
    "    grid_dict = get_grid_dict(city_path, city_name)\n",
    "    # grid_dict = {'x,y': ID}\n",
    "    trans_filename = os.path.join(city_path, \"transfer.csv\")\n",
    "    output_file = os.path.join(output_path, \"%s_transfer.csv\" % (city_name))\n",
    "    with open(trans_filename, 'r') as f, open(output_file, 'w') as writer:\n",
    "        for line in f:\n",
    "            items = line.strip().split(',')\n",
    "            start_axis = \",\".join(items[1:3])\n",
    "            end_axis = \",\".join(items[3:5])\n",
    "            index = items[5]\n",
    "            try:\n",
    "                start_ID = grid_dict[start_axis]\n",
    "                end_ID = grid_dict[end_axis] \n",
    "            except KeyError: # remove no ID axis\n",
    "                continue\n",
    "\n",
    "            writer.write(\"%s,%s,%s,%s\\n\" % (items[0], start_ID, end_ID, index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "coord2ID('./dataset/train_data', 'A', './dataset')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>hour</th>\n",
       "      <th>s_region</th>\n",
       "      <th>e_region</th>\n",
       "      <th>index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>A_0</td>\n",
       "      <td>A_0</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>A_0</td>\n",
       "      <td>A_0</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>A_0</td>\n",
       "      <td>A_0</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>A_0</td>\n",
       "      <td>A_0</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>A_0</td>\n",
       "      <td>A_0</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   hour s_region e_region  index\n",
       "0     0      A_0      A_0    0.1\n",
       "1     0      A_0      A_0    0.2\n",
       "2     0      A_0      A_0    0.1\n",
       "3     0      A_0      A_0    0.1\n",
       "4     0      A_0      A_0    0.1"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看得到的每小时区域人群迁移指数\n",
    "pd.read_csv('dataset/A_transfer.csv', header=None, names=['hour', 's_region', 'e_region', 'index']).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 求和计算每日市内区域人群迁移指数\n",
    "def calc_index_in_one_day(data_path, city_name):\n",
    "    trans_filename = os.path.join(data_path, \"%s_transfer.csv\" % (city_name))\n",
    "    transfer = pd.read_csv(trans_filename, \n",
    "            header=None,\n",
    "            names=['hour', 's_region', 'e_region', 'index'])\n",
    "        \n",
    "    df = transfer.groupby(['s_region', 'e_region'])['index'].sum().reset_index()\n",
    "    df = df[['s_region', 'e_region', 'index']]\n",
    "    #  df = df.T\n",
    "    #  df_list.append(df)\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>s_region</th>\n",
       "      <th>e_region</th>\n",
       "      <th>index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_0</td>\n",
       "      <td>187.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_1</td>\n",
       "      <td>0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_10</td>\n",
       "      <td>0.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_100</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_108</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_11</td>\n",
       "      <td>0.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_110</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_111</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_112</td>\n",
       "      <td>0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_116</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_117</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_12</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_13</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_14</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_15</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_16</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_17</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_18</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_19</td>\n",
       "      <td>0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_2</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_20</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_23</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_25</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_26</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_29</td>\n",
       "      <td>2.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_30</td>\n",
       "      <td>0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_31</td>\n",
       "      <td>0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_32</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_33</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>A_0</td>\n",
       "      <td>A_34</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13026</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_72</td>\n",
       "      <td>1.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13027</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_73</td>\n",
       "      <td>0.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13028</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_74</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13029</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_75</td>\n",
       "      <td>1.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13030</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_76</td>\n",
       "      <td>1.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13031</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_77</td>\n",
       "      <td>1.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13032</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_78</td>\n",
       "      <td>5.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13033</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_79</td>\n",
       "      <td>6.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13034</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_8</td>\n",
       "      <td>3.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13035</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_80</td>\n",
       "      <td>5.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13036</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_81</td>\n",
       "      <td>9.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13037</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_82</td>\n",
       "      <td>3.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13038</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_83</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13039</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_84</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13040</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_85</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13041</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_86</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13042</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_87</td>\n",
       "      <td>26.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13043</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_88</td>\n",
       "      <td>32.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13044</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_89</td>\n",
       "      <td>7.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13045</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_9</td>\n",
       "      <td>1.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13046</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_90</td>\n",
       "      <td>1.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13047</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_91</td>\n",
       "      <td>0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13048</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_92</td>\n",
       "      <td>69.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13049</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_93</td>\n",
       "      <td>119.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13050</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_94</td>\n",
       "      <td>50.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13051</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_95</td>\n",
       "      <td>3.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13052</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_96</td>\n",
       "      <td>0.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13053</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_97</td>\n",
       "      <td>1.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13054</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_98</td>\n",
       "      <td>135.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13055</th>\n",
       "      <td>A_99</td>\n",
       "      <td>A_99</td>\n",
       "      <td>153.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>13056 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      s_region e_region  index\n",
       "0          A_0      A_0  187.5\n",
       "1          A_0      A_1    0.3\n",
       "2          A_0     A_10    0.8\n",
       "3          A_0    A_100    0.2\n",
       "4          A_0    A_108    0.4\n",
       "5          A_0     A_11    0.7\n",
       "6          A_0    A_110    0.1\n",
       "7          A_0    A_111    0.1\n",
       "8          A_0    A_112    0.3\n",
       "9          A_0    A_116    0.1\n",
       "10         A_0    A_117    0.1\n",
       "11         A_0     A_12    0.1\n",
       "12         A_0     A_13    0.4\n",
       "13         A_0     A_14    0.2\n",
       "14         A_0     A_15    0.2\n",
       "15         A_0     A_16    0.4\n",
       "16         A_0     A_17    0.1\n",
       "17         A_0     A_18    0.5\n",
       "18         A_0     A_19    0.3\n",
       "19         A_0      A_2    0.2\n",
       "20         A_0     A_20    0.6\n",
       "21         A_0     A_23    0.2\n",
       "22         A_0     A_25    0.1\n",
       "23         A_0     A_26    0.2\n",
       "24         A_0     A_29    2.7\n",
       "25         A_0     A_30    0.3\n",
       "26         A_0     A_31    0.3\n",
       "27         A_0     A_32    0.1\n",
       "28         A_0     A_33    0.1\n",
       "29         A_0     A_34    0.2\n",
       "...        ...      ...    ...\n",
       "13026     A_99     A_72    1.2\n",
       "13027     A_99     A_73    0.7\n",
       "13028     A_99     A_74    0.6\n",
       "13029     A_99     A_75    1.1\n",
       "13030     A_99     A_76    1.1\n",
       "13031     A_99     A_77    1.7\n",
       "13032     A_99     A_78    5.4\n",
       "13033     A_99     A_79    6.6\n",
       "13034     A_99      A_8    3.4\n",
       "13035     A_99     A_80    5.7\n",
       "13036     A_99     A_81    9.3\n",
       "13037     A_99     A_82    3.8\n",
       "13038     A_99     A_83    5.0\n",
       "13039     A_99     A_84    2.0\n",
       "13040     A_99     A_85    0.4\n",
       "13041     A_99     A_86    0.1\n",
       "13042     A_99     A_87   26.8\n",
       "13043     A_99     A_88   32.7\n",
       "13044     A_99     A_89    7.6\n",
       "13045     A_99      A_9    1.4\n",
       "13046     A_99     A_90    1.2\n",
       "13047     A_99     A_91    0.3\n",
       "13048     A_99     A_92   69.0\n",
       "13049     A_99     A_93  119.5\n",
       "13050     A_99     A_94   50.9\n",
       "13051     A_99     A_95    3.9\n",
       "13052     A_99     A_96    0.8\n",
       "13053     A_99     A_97    1.2\n",
       "13054     A_99     A_98  135.4\n",
       "13055     A_99     A_99  153.0\n",
       "\n",
       "[13056 rows x 3 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calc_index_in_one_day('./dataset', 'A')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 计算城市迁移指数\n",
    "每个城市的`migration.csv`记录了从该城市出发、到达该城市的人流量指数，可以通过统计计算每日到达指定城市的人流量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def process_city_migration(data_path, city_name):\n",
    "    filename = os.path.join(data_path, \"city_%s\" % city_name, \"migration.csv\")\n",
    "    migration = pd.read_csv(filename, \n",
    "                            sep=',', \n",
    "                            header=None,\n",
    "                            names=['date', 's_city', 'e_city', city_name])\n",
    "\n",
    "    # only use moving in \"city\" data, ignore moving out data\n",
    "    df = migration[migration.e_city == city_name]\n",
    "    df = df[[\"date\", city_name]]\n",
    "\n",
    "    # calculate total move in data of \"city\"\n",
    "    df = df.groupby('date')[city_name].sum().reset_index()\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>A</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>21200501</td>\n",
       "      <td>0.811620</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>21200502</td>\n",
       "      <td>0.742641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21200503</td>\n",
       "      <td>0.964937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>21200504</td>\n",
       "      <td>0.771767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>21200505</td>\n",
       "      <td>0.727024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>21200506</td>\n",
       "      <td>1.101211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>21200507</td>\n",
       "      <td>0.750903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>21200508</td>\n",
       "      <td>1.004562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>21200509</td>\n",
       "      <td>0.887760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>21200510</td>\n",
       "      <td>0.890514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>21200511</td>\n",
       "      <td>1.074288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>21200512</td>\n",
       "      <td>0.802903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>21200513</td>\n",
       "      <td>0.715456</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>21200514</td>\n",
       "      <td>0.713415</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>21200515</td>\n",
       "      <td>0.724205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>21200516</td>\n",
       "      <td>0.846061</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>21200517</td>\n",
       "      <td>0.784372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>21200518</td>\n",
       "      <td>1.017717</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>21200519</td>\n",
       "      <td>0.842400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>21200520</td>\n",
       "      <td>0.751421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>21200521</td>\n",
       "      <td>1.040493</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>21200522</td>\n",
       "      <td>0.796489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>21200523</td>\n",
       "      <td>1.011236</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>21200524</td>\n",
       "      <td>0.708232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>21200525</td>\n",
       "      <td>0.730554</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>21200526</td>\n",
       "      <td>0.740178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>21200527</td>\n",
       "      <td>0.714582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>21200528</td>\n",
       "      <td>0.935291</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>21200529</td>\n",
       "      <td>0.759197</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>21200530</td>\n",
       "      <td>1.102637</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>21200531</td>\n",
       "      <td>0.955054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>21200601</td>\n",
       "      <td>0.733666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>21200602</td>\n",
       "      <td>0.669124</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>21200603</td>\n",
       "      <td>0.669902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>21200604</td>\n",
       "      <td>0.684385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>21200605</td>\n",
       "      <td>0.541469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>21200606</td>\n",
       "      <td>0.664135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>21200607</td>\n",
       "      <td>0.669902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>21200608</td>\n",
       "      <td>0.508906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>21200609</td>\n",
       "      <td>0.139514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>21200610</td>\n",
       "      <td>0.684385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>21200611</td>\n",
       "      <td>0.379826</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>21200612</td>\n",
       "      <td>0.770699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>21200613</td>\n",
       "      <td>0.508906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>21200614</td>\n",
       "      <td>0.541469</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        date         A\n",
       "0   21200501  0.811620\n",
       "1   21200502  0.742641\n",
       "2   21200503  0.964937\n",
       "3   21200504  0.771767\n",
       "4   21200505  0.727024\n",
       "5   21200506  1.101211\n",
       "6   21200507  0.750903\n",
       "7   21200508  1.004562\n",
       "8   21200509  0.887760\n",
       "9   21200510  0.890514\n",
       "10  21200511  1.074288\n",
       "11  21200512  0.802903\n",
       "12  21200513  0.715456\n",
       "13  21200514  0.713415\n",
       "14  21200515  0.724205\n",
       "15  21200516  0.846061\n",
       "16  21200517  0.784372\n",
       "17  21200518  1.017717\n",
       "18  21200519  0.842400\n",
       "19  21200520  0.751421\n",
       "20  21200521  1.040493\n",
       "21  21200522  0.796489\n",
       "22  21200523  1.011236\n",
       "23  21200524  0.708232\n",
       "24  21200525  0.730554\n",
       "25  21200526  0.740178\n",
       "26  21200527  0.714582\n",
       "27  21200528  0.935291\n",
       "28  21200529  0.759197\n",
       "29  21200530  1.102637\n",
       "30  21200531  0.955054\n",
       "31  21200601  0.733666\n",
       "32  21200602  0.669124\n",
       "33  21200603  0.669902\n",
       "34  21200604  0.684385\n",
       "35  21200605  0.541469\n",
       "36  21200606  0.664135\n",
       "37  21200607  0.669902\n",
       "38  21200608  0.508906\n",
       "39  21200609  0.139514\n",
       "40  21200610  0.684385\n",
       "41  21200611  0.379826\n",
       "42  21200612  0.770699\n",
       "43  21200613  0.508906\n",
       "44  21200614  0.541469"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 计算每日到达A地的人流量\n",
    "process_city_migration('./dataset/train_data', 'A')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def migration_process(data_path, city_list, output_path):\n",
    "    for city_name in city_list:\n",
    "        coord2ID(data_path, city_name, output_path)\n",
    "        transfer = calc_index_in_one_day(output_path, city_name)\n",
    "        migration = process_city_migration(data_path, city_name)\n",
    "\n",
    "        df_list = []\n",
    "        for i in range(len(migration)):\n",
    "            df = transfer.copy()\n",
    "            date = migration.date[i]\n",
    "            index = migration[city_name][i]\n",
    "            # 这里通过将到达城市的人流量指数与市内区域人流量指数相乘，得到区域人流量指数\n",
    "            df['index'] = df['index'] * index\n",
    "            df['date'] = date\n",
    "            df = df[['date', 's_region', 'e_region', 'index']]\n",
    "            # 按日新增区域人流量统计数据\n",
    "            df_list.append(df)\n",
    "\n",
    "        df = pd.concat(df_list, axis=0)\n",
    "        # 得到每个城市最终迁移人流量指数\n",
    "        df.to_csv(os.path.join(output_path, '%s_migration.csv' % city_name), \n",
    "                header=None,\n",
    "                index=None,\n",
    "                float_format = '%.4f')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "migration_process('./dataset/train_data', [\"A\", \"B\", \"C\", \"D\", \"E\"], './dataset')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>21200501</td>\n",
       "      <td>A_0</td>\n",
       "      <td>A_0</td>\n",
       "      <td>152.1787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>21200501</td>\n",
       "      <td>A_0</td>\n",
       "      <td>A_1</td>\n",
       "      <td>0.2435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21200501</td>\n",
       "      <td>A_0</td>\n",
       "      <td>A_10</td>\n",
       "      <td>0.6493</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>21200501</td>\n",
       "      <td>A_0</td>\n",
       "      <td>A_100</td>\n",
       "      <td>0.1623</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>21200501</td>\n",
       "      <td>A_0</td>\n",
       "      <td>A_108</td>\n",
       "      <td>0.3246</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0    1      2         3\n",
       "0  21200501  A_0    A_0  152.1787\n",
       "1  21200501  A_0    A_1    0.2435\n",
       "2  21200501  A_0   A_10    0.6493\n",
       "3  21200501  A_0  A_100    0.1623\n",
       "4  21200501  A_0  A_108    0.3246"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('dataset/A_migration.csv', header=None).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 计算邻接矩阵"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def adj_matrix_process(data_path, city_list, region_nums, output_path):\n",
    "    total_region_num = np.sum(region_nums)\n",
    "    adj_matrix = np.zeros((total_region_num, total_region_num))\n",
    "\n",
    "    offset = 0\n",
    "    for i, city in enumerate(city_list):\n",
    "        filename = os.path.join(output_path, \"%s_migration.csv\" % city)\n",
    "        migration = pd.read_csv(filename, \n",
    "                                sep=',', \n",
    "                                header=None,\n",
    "                                names=['date', 's_region', 'e_region', 'index'])\n",
    "        # 生成每个城市的初始邻接矩阵，比如A城市是有118个区域，shape就是(118, 118)\n",
    "        matrix = np.zeros((region_nums[i], region_nums[i]))\n",
    "        # pdb.set_trace()\n",
    "        # 对区域编码进行排序，官方baseline的写法会把10,100排到2,3,4等前面\n",
    "        # order = sorted(range(region_nums[i]), key=lambda x:str(x))\n",
    "        order = sorted(list(range(region_nums[i])))\n",
    "        for j, idx in enumerate(order):\n",
    "            # 拼接目标区域的标准名称：城市名称+区域ID\n",
    "            target_region = \"%s_%d\" % (city, idx)\n",
    "            # only use moving in \"city\" data, ignore moving out data\n",
    "            # 只用到迁入城市的人流量，不考虑迁出的问题\n",
    "            df = migration[migration['e_region'] == target_region]\n",
    "\n",
    "            # 计算得到每个区域迁入的平均人流量\n",
    "            df = df.groupby('s_region')['index'].mean().reset_index()\n",
    "            #  res = df['index'].values.reshape(-1)\n",
    "            for k, o in enumerate(order):\n",
    "                s_region_id = \"%s_%d\" % (city, o)\n",
    "                try:\n",
    "                    # 取出来自指定出发地的平均人流量数据\n",
    "                    value = df[df['s_region'] == s_region_id]['index'].values[0]\n",
    "                except:\n",
    "                    value = 0.0\n",
    "                if s_region_id == target_region:\n",
    "                    value = 0.0\n",
    "                # 给邻接矩阵该位置的元素赋值\n",
    "                matrix[j, k] = value\n",
    "\n",
    "        # merge two adj_matrix\n",
    "        # 把不同城市的邻接矩阵拼起来，形成最终的大的邻接矩阵\n",
    "        adj_matrix[offset:(offset + region_nums[i]), offset:(offset + region_nums[i])] = matrix\n",
    "        offset += region_nums[i]\n",
    "    # 这里调整了一下，保存为csv格式\n",
    "    file_to_save = os.path.join(output_path, 'adj_matrix.csv')\n",
    "    print(\"saving result to %s\" % file_to_save)\n",
    "    # np.save(file_to_save, adj_matrix)\n",
    "    np.savetxt(file_to_save, adj_matrix, delimiter=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving result to ./dataset/adj_matrix.csv\n"
     ]
    }
   ],
   "source": [
    "adj_matrix_process('./dataset/train_data', [\"A\", \"B\", \"C\", \"D\", \"E\"], [118, 30, 135, 75, 34], './dataset')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>382</th>\n",
       "      <th>383</th>\n",
       "      <th>384</th>\n",
       "      <th>385</th>\n",
       "      <th>386</th>\n",
       "      <th>387</th>\n",
       "      <th>388</th>\n",
       "      <th>389</th>\n",
       "      <th>390</th>\n",
       "      <th>391</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.076853</td>\n",
       "      <td>0.307436</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.384284</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.230571</td>\n",
       "      <td>0.230571</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.230571</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.230571</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.153711</td>\n",
       "      <td>0.230571</td>\n",
       "      <td>0.076853</td>\n",
       "      <td>0.999138</td>\n",
       "      <td>0.076853</td>\n",
       "      <td>0.153711</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.153711</td>\n",
       "      <td>0.307436</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.996533</td>\n",
       "      <td>13.219304</td>\n",
       "      <td>0.537987</td>\n",
       "      <td>0.845416</td>\n",
       "      <td>1.767704</td>\n",
       "      <td>1.460271</td>\n",
       "      <td>4.611380</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.535398</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>8.377353</td>\n",
       "      <td>0.076853</td>\n",
       "      <td>6.993933</td>\n",
       "      <td>1.844553</td>\n",
       "      <td>1.152851</td>\n",
       "      <td>5.379951</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.230571</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>14.372151</td>\n",
       "      <td>8.069929</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.922280</td>\n",
       "      <td>2.613129</td>\n",
       "      <td>7.685642</td>\n",
       "      <td>4.841956</td>\n",
       "      <td>18.906689</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 392 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        0         1          2         3          4         5         6    \\\n",
       "0  0.000000  0.076853   0.307436  0.000000   0.384284  0.000000  0.230571   \n",
       "1  0.230571  0.000000   0.153711  0.230571   0.076853  0.999138  0.076853   \n",
       "2  0.153711  0.307436   0.000000  3.996533  13.219304  0.537987  0.845416   \n",
       "3  0.000000  0.000000   3.535398  0.000000   8.377353  0.076853  6.993933   \n",
       "4  0.230571  0.000000  14.372151  8.069929   0.000000  0.922280  2.613129   \n",
       "\n",
       "        7         8          9   ...   382  383  384  385  386  387  388  389  \\\n",
       "0  0.230571  0.000000   0.230571 ...   0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0   \n",
       "1  0.153711  0.000000   0.000000 ...   0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0   \n",
       "2  1.767704  1.460271   4.611380 ...   0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0   \n",
       "3  1.844553  1.152851   5.379951 ...   0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0   \n",
       "4  7.685642  4.841956  18.906689 ...   0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0   \n",
       "\n",
       "   390  391  \n",
       "0  0.0  0.0  \n",
       "1  0.0  0.0  \n",
       "2  0.0  0.0  \n",
       "3  0.0  0.0  \n",
       "4  0.0  0.0  \n",
       "\n",
       "[5 rows x 392 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('dataset/adj_matrix.csv',header=None).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 感染人数处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>city</th>\n",
       "      <th>region</th>\n",
       "      <th>date</th>\n",
       "      <th>infect</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "      <td>21200501</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "      <td>21200502</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "      <td>21200503</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "      <td>21200504</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A</td>\n",
       "      <td>0</td>\n",
       "      <td>21200505</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  city  region      date  infect\n",
       "0    A       0  21200501       0\n",
       "1    A       0  21200502       0\n",
       "2    A       0  21200503       0\n",
       "3    A       0  21200504       0\n",
       "4    A       0  21200505       0"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('dataset/train_data/city_A/infection.csv', header=None, names=[\"city\", \"region\", \"date\", \"infect\"]).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def infection_process(data_path, city_list, region_nums, output_path):\n",
    "    res = []\n",
    "    region_name_list = []\n",
    "    for i, city in enumerate(city_list):\n",
    "        filename = os.path.join(data_path, \"city_%s\" % city, \"infection.csv\")\n",
    "        migration = pd.read_csv(filename, \n",
    "                                sep=',', \n",
    "                                header=None,\n",
    "                                names=[\"city\", \"region\", \"date\", \"infect\"])\n",
    "\n",
    "        # order = sorted(range(region_nums[i]), key=lambda x:str(x))\n",
    "        order = sorted(list(range(region_nums[i])))\n",
    "        for j, idx in enumerate(order):\n",
    "            target_region = idx #str(idx)\n",
    "            # pdb.set_trace()\n",
    "            # 区域每天感染人数\n",
    "            df = migration[migration['region'] == target_region].reset_index(drop=True)\n",
    "            if i == 0 and j == 0:\n",
    "                # 第一个区域要把日期给进去\n",
    "                df = df[['date', 'infect']]\n",
    "            else:\n",
    "                df = df[['infect']]\n",
    "\n",
    "            df = df.rename(columns={'infect': '%s_%d' % (city, idx)})\n",
    "            region_name_list.append(\"%s_%d\" % (city, idx))\n",
    "\n",
    "            res.append(df)\n",
    "    df = pd.concat(res, axis=1)\n",
    "    # 最终形成城市+区域ID形式的感染人数大宽表\n",
    "    file_to_save = os.path.join(output_path, \"infection.csv\")\n",
    "    print(\"saving result to %s\" % file_to_save)\n",
    "    # format: [date, A, B, C, D, E]\n",
    "    df.to_csv(file_to_save, index=False)\n",
    "\n",
    "    region_name_file = os.path.join(output_path, \"region_names.txt\")\n",
    "    with open(region_name_file, 'w') as f:\n",
    "        names = ' '.join(region_name_list)\n",
    "        f.write(names + '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving result to ./dataset/infection.csv\n"
     ]
    }
   ],
   "source": [
    "infection_process('./dataset/train_data', [\"A\", \"B\", \"C\", \"D\", \"E\"], [118, 30, 135, 75, 34], './dataset')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>A_0</th>\n",
       "      <th>A_1</th>\n",
       "      <th>A_2</th>\n",
       "      <th>A_3</th>\n",
       "      <th>A_4</th>\n",
       "      <th>A_5</th>\n",
       "      <th>A_6</th>\n",
       "      <th>A_7</th>\n",
       "      <th>A_8</th>\n",
       "      <th>...</th>\n",
       "      <th>E_24</th>\n",
       "      <th>E_25</th>\n",
       "      <th>E_26</th>\n",
       "      <th>E_27</th>\n",
       "      <th>E_28</th>\n",
       "      <th>E_29</th>\n",
       "      <th>E_30</th>\n",
       "      <th>E_31</th>\n",
       "      <th>E_32</th>\n",
       "      <th>E_33</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>21200501</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>21200502</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21200503</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>21200504</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>21200505</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 393 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       date  A_0  A_1  A_2  A_3  A_4  A_5  A_6  A_7  A_8  ...   E_24  E_25  \\\n",
       "0  21200501    0    0    0    0    0    0    0    0    0  ...      0     0   \n",
       "1  21200502    0    0    0    0    0    0    0    0    0  ...      0     0   \n",
       "2  21200503    0    0    0    0    0    0    0    0    0  ...      0     0   \n",
       "3  21200504    0    0    0    0    0    0    0    0    0  ...      0     0   \n",
       "4  21200505    0    0    0    0    0    0    0    0    0  ...      0     0   \n",
       "\n",
       "   E_26  E_27  E_28  E_29  E_30  E_31  E_32  E_33  \n",
       "0     0     0     0     0     0     0     0     0  \n",
       "1     0     0     0     0     0     0     0     0  \n",
       "2     0     0     0     0     0     0     0     0  \n",
       "3     0     0     0     0     0     0     0     0  \n",
       "4     0     0     0     0     0     0     0     0  \n",
       "\n",
       "[5 rows x 393 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('dataset/infection.csv').head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 迁移人流量处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def region_migration_process(data_path, city_list, region_nums, output_path):\n",
    "    res = []\n",
    "    # 这里和感染人数处理类似\n",
    "    for i, city in enumerate(city_list):\n",
    "        filename = os.path.join(output_path, \"%s_migration.csv\" % city)\n",
    "        migration = pd.read_csv(filename, \n",
    "                                sep=',', \n",
    "                                header=None,\n",
    "                                names=['date', 's_region', 'e_region', 'index'])\n",
    "\n",
    "        # order = sorted(range(region_nums[i]), key=lambda x:str(x))\n",
    "        order = sorted(list(range(region_nums[i])))\n",
    "        for j, idx in enumerate(order):\n",
    "            target_region = \"%s_%d\" % (city, idx)\n",
    "            df = migration[migration['e_region'] == target_region]\n",
    "\n",
    "            df = df.groupby('date')['index'].sum().reset_index()\n",
    "\n",
    "            if i == 0 and j == 0:\n",
    "                df = df[['date', 'index']]\n",
    "            else:\n",
    "                df = df[['index']]\n",
    "\n",
    "            df = df.rename(columns={'index': target_region})\n",
    "\n",
    "            res.append(df)\n",
    "    # 最终形成城市+区域ID形式的迁移人流量大宽表\n",
    "    df = pd.concat(res, axis=1)\n",
    "\n",
    "    file_to_save = os.path.join(output_path, \"region_migration.csv\")\n",
    "    print(\"saving result to %s\" % file_to_save)\n",
    "    # format: [date, A, B, C, D, E]\n",
    "    df.to_csv(file_to_save, index=False, float_format = '%.2f')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving result to ./dataset/region_migration.csv\n"
     ]
    }
   ],
   "source": [
    "region_migration_process('./dataset/train_data', [\"A\", \"B\", \"C\", \"D\", \"E\"], [118, 30, 135, 75, 34], './dataset')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>A_0</th>\n",
       "      <th>A_1</th>\n",
       "      <th>A_2</th>\n",
       "      <th>A_3</th>\n",
       "      <th>A_4</th>\n",
       "      <th>A_5</th>\n",
       "      <th>A_6</th>\n",
       "      <th>A_7</th>\n",
       "      <th>A_8</th>\n",
       "      <th>...</th>\n",
       "      <th>E_24</th>\n",
       "      <th>E_25</th>\n",
       "      <th>E_26</th>\n",
       "      <th>E_27</th>\n",
       "      <th>E_28</th>\n",
       "      <th>E_29</th>\n",
       "      <th>E_30</th>\n",
       "      <th>E_31</th>\n",
       "      <th>E_32</th>\n",
       "      <th>E_33</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>21200501</td>\n",
       "      <td>171.82</td>\n",
       "      <td>200.06</td>\n",
       "      <td>231.88</td>\n",
       "      <td>209.15</td>\n",
       "      <td>403.13</td>\n",
       "      <td>179.94</td>\n",
       "      <td>204.45</td>\n",
       "      <td>897.98</td>\n",
       "      <td>1156.07</td>\n",
       "      <td>...</td>\n",
       "      <td>63.98</td>\n",
       "      <td>50.18</td>\n",
       "      <td>50.74</td>\n",
       "      <td>64.47</td>\n",
       "      <td>21.42</td>\n",
       "      <td>7.04</td>\n",
       "      <td>11.47</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9.93</td>\n",
       "      <td>3.14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>21200502</td>\n",
       "      <td>157.22</td>\n",
       "      <td>183.06</td>\n",
       "      <td>212.17</td>\n",
       "      <td>191.38</td>\n",
       "      <td>368.87</td>\n",
       "      <td>164.64</td>\n",
       "      <td>187.07</td>\n",
       "      <td>821.66</td>\n",
       "      <td>1057.82</td>\n",
       "      <td>...</td>\n",
       "      <td>56.44</td>\n",
       "      <td>44.27</td>\n",
       "      <td>44.76</td>\n",
       "      <td>56.87</td>\n",
       "      <td>18.90</td>\n",
       "      <td>6.21</td>\n",
       "      <td>10.12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.76</td>\n",
       "      <td>2.77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21200503</td>\n",
       "      <td>204.28</td>\n",
       "      <td>237.86</td>\n",
       "      <td>275.68</td>\n",
       "      <td>248.66</td>\n",
       "      <td>479.28</td>\n",
       "      <td>213.93</td>\n",
       "      <td>243.07</td>\n",
       "      <td>1067.61</td>\n",
       "      <td>1374.46</td>\n",
       "      <td>...</td>\n",
       "      <td>70.30</td>\n",
       "      <td>55.13</td>\n",
       "      <td>55.75</td>\n",
       "      <td>70.83</td>\n",
       "      <td>23.54</td>\n",
       "      <td>7.73</td>\n",
       "      <td>12.60</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.91</td>\n",
       "      <td>3.45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>21200504</td>\n",
       "      <td>163.38</td>\n",
       "      <td>190.24</td>\n",
       "      <td>220.49</td>\n",
       "      <td>198.89</td>\n",
       "      <td>383.34</td>\n",
       "      <td>171.10</td>\n",
       "      <td>194.41</td>\n",
       "      <td>853.88</td>\n",
       "      <td>1099.30</td>\n",
       "      <td>...</td>\n",
       "      <td>58.88</td>\n",
       "      <td>46.18</td>\n",
       "      <td>46.69</td>\n",
       "      <td>59.33</td>\n",
       "      <td>19.72</td>\n",
       "      <td>6.47</td>\n",
       "      <td>10.55</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9.14</td>\n",
       "      <td>2.89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>21200505</td>\n",
       "      <td>153.91</td>\n",
       "      <td>179.21</td>\n",
       "      <td>207.71</td>\n",
       "      <td>187.35</td>\n",
       "      <td>361.11</td>\n",
       "      <td>161.18</td>\n",
       "      <td>183.14</td>\n",
       "      <td>804.38</td>\n",
       "      <td>1035.57</td>\n",
       "      <td>...</td>\n",
       "      <td>57.64</td>\n",
       "      <td>45.20</td>\n",
       "      <td>45.71</td>\n",
       "      <td>58.07</td>\n",
       "      <td>19.30</td>\n",
       "      <td>6.34</td>\n",
       "      <td>10.33</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.95</td>\n",
       "      <td>2.83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 393 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       date     A_0     A_1     A_2     A_3     A_4     A_5     A_6      A_7  \\\n",
       "0  21200501  171.82  200.06  231.88  209.15  403.13  179.94  204.45   897.98   \n",
       "1  21200502  157.22  183.06  212.17  191.38  368.87  164.64  187.07   821.66   \n",
       "2  21200503  204.28  237.86  275.68  248.66  479.28  213.93  243.07  1067.61   \n",
       "3  21200504  163.38  190.24  220.49  198.89  383.34  171.10  194.41   853.88   \n",
       "4  21200505  153.91  179.21  207.71  187.35  361.11  161.18  183.14   804.38   \n",
       "\n",
       "       A_8  ...    E_24   E_25   E_26   E_27   E_28  E_29   E_30  E_31   E_32  \\\n",
       "0  1156.07  ...   63.98  50.18  50.74  64.47  21.42  7.04  11.47   NaN   9.93   \n",
       "1  1057.82  ...   56.44  44.27  44.76  56.87  18.90  6.21  10.12   NaN   8.76   \n",
       "2  1374.46  ...   70.30  55.13  55.75  70.83  23.54  7.73  12.60   NaN  10.91   \n",
       "3  1099.30  ...   58.88  46.18  46.69  59.33  19.72  6.47  10.55   NaN   9.14   \n",
       "4  1035.57  ...   57.64  45.20  45.71  58.07  19.30  6.34  10.33   NaN   8.95   \n",
       "\n",
       "   E_33  \n",
       "0  3.14  \n",
       "1  2.77  \n",
       "2  3.45  \n",
       "3  2.89  \n",
       "4  2.83  \n",
       "\n",
       "[5 rows x 393 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('dataset/region_migration.csv').head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] 2020-06-26 14:28:43,150 [     main.py:  219]:\tNamespace(Ks=3, Kt=4, adj_mat_file='./dataset/adj_matrix.csv', batch_size=1, blocks=[[1, 16, 32], [32, 16, 64]], city_num=392, epochs=10, feat_dim=1, input_file='./dataset/region_migration.csv', keep_prob=1.0, label_file='./dataset/infection.csv', lr=0.005, n_his=20, n_pred=1, opt='ADAM', output_path='../outputs/', region_names_file='./dataset/region_names.txt', save=5, seed=1, submit_file='./dataset/train_data/submission.csv', test_num=1, use_cuda=False, val_num=3)\n",
      "[INFO] 2020-06-26 14:28:44,674 [     main.py:   39]:\tnum examples: 26\n",
      "[INFO] 2020-06-26 14:28:44,675 [     main.py:   45]:\tTrain examples: 22\n",
      "[INFO] 2020-06-26 14:28:44,676 [     main.py:   46]:\tTest examples: 1\n",
      "[INFO] 2020-06-26 14:28:44,676 [     main.py:   50]:\tValid examples: 3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "region migration:         date      A_0      A_1      A_2      A_3      A_4      A_5      A_6  \\\n",
      "0  21200501  0.17182  0.20006  0.23188  0.20915  0.40313  0.17994  0.20445   \n",
      "1  21200502  0.15722  0.18306  0.21217  0.19138  0.36887  0.16464  0.18707   \n",
      "2  21200503  0.20428  0.23786  0.27568  0.24866  0.47928  0.21393  0.24307   \n",
      "3  21200504  0.16338  0.19024  0.22049  0.19889  0.38334  0.17110  0.19441   \n",
      "4  21200505  0.15391  0.17921  0.20771  0.18735  0.36111  0.16118  0.18314   \n",
      "\n",
      "       A_7      A_8   ...        E_24     E_25     E_26     E_27     E_28  \\\n",
      "0  0.89798  1.15607   ...     0.06398  0.05018  0.05074  0.06447  0.02142   \n",
      "1  0.82166  1.05782   ...     0.05644  0.04427  0.04476  0.05687  0.01890   \n",
      "2  1.06761  1.37446   ...     0.07030  0.05513  0.05575  0.07083  0.02354   \n",
      "3  0.85388  1.09930   ...     0.05888  0.04618  0.04669  0.05933  0.01972   \n",
      "4  0.80438  1.03557   ...     0.05764  0.04520  0.04571  0.05807  0.01930   \n",
      "\n",
      "      E_29     E_30  E_31     E_32     E_33  \n",
      "0  0.00704  0.01147   0.0  0.00993  0.00314  \n",
      "1  0.00621  0.01012   0.0  0.00876  0.00277  \n",
      "2  0.00773  0.01260   0.0  0.01091  0.00345  \n",
      "3  0.00647  0.01055   0.0  0.00914  0.00289  \n",
      "4  0.00634  0.01033   0.0  0.00895  0.00283  \n",
      "\n",
      "[5 rows x 393 columns]\n",
      "infect:         date  A_0  A_1  A_2  A_3  A_4  A_5  A_6  A_7  A_8  ...   E_24  E_25  \\\n",
      "0  21200501  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...    0.0   0.0   \n",
      "1  21200502  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...    0.0   0.0   \n",
      "2  21200503  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...    0.0   0.0   \n",
      "3  21200504  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...    0.0   0.0   \n",
      "4  21200505  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...    0.0   0.0   \n",
      "\n",
      "   E_26  E_27  E_28  E_29  E_30  E_31  E_32  E_33  \n",
      "0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
      "1   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
      "2   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
      "3   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
      "4   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
      "\n",
      "[5 rows x 393 columns]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] 2020-06-26 14:29:22,067 [     main.py:  103]:\tepoch 1 | step 5 | loss 5141.372070\n",
      "[INFO] 2020-06-26 14:29:57,194 [     main.py:  103]:\tepoch 1 | step 10 | loss 14939.403320\n",
      "[INFO] 2020-06-26 14:30:12,754 [     main.py:  114]:\tvalid result: | rmsle 0.3194172598309787 \n",
      "[INFO] 2020-06-26 14:32:13,501 [     main.py:  103]:\tepoch 1 | step 15 | loss 45625.109375\n",
      "[INFO] 2020-06-26 14:32:48,288 [     main.py:  103]:\tepoch 1 | step 20 | loss 2329.699219\n",
      "[INFO] 2020-06-26 14:33:03,864 [     main.py:  114]:\tvalid result: | rmsle 0.3262588392553634 \n",
      "[INFO] 2020-06-26 14:33:38,870 [     main.py:  103]:\tepoch 2 | step 25 | loss 1218.109741\n",
      "[INFO] 2020-06-26 14:34:14,048 [     main.py:  103]:\tepoch 2 | step 30 | loss 3981.177734\n",
      "[INFO] 2020-06-26 14:34:30,069 [     main.py:  114]:\tvalid result: | rmsle 0.31293732133227786 \n",
      "[INFO] 2020-06-26 14:36:28,008 [     main.py:  103]:\tepoch 2 | step 35 | loss 51102.667969\n",
      "[INFO] 2020-06-26 14:37:02,589 [     main.py:  103]:\tepoch 2 | step 40 | loss 4952.343262\n",
      "[INFO] 2020-06-26 14:37:18,472 [     main.py:  114]:\tvalid result: | rmsle 0.3501166642559801 \n",
      "[INFO] 2020-06-26 14:37:53,013 [     main.py:  103]:\tepoch 3 | step 45 | loss 1652.163452\n",
      "[INFO] 2020-06-26 14:38:27,370 [     main.py:  103]:\tepoch 3 | step 50 | loss 1507.490601\n",
      "[INFO] 2020-06-26 14:38:43,138 [     main.py:  114]:\tvalid result: | rmsle 0.331668068887688 \n",
      "[INFO] 2020-06-26 14:39:17,488 [     main.py:  103]:\tepoch 3 | step 55 | loss 21587.957031\n",
      "[INFO] 2020-06-26 14:39:52,009 [     main.py:  103]:\tepoch 3 | step 60 | loss 33287.515625\n",
      "[INFO] 2020-06-26 14:40:07,656 [     main.py:  114]:\tvalid result: | rmsle 0.44680587784508113 \n",
      "[INFO] 2020-06-26 14:40:41,938 [     main.py:  103]:\tepoch 3 | step 65 | loss 1946.974976\n",
      "[INFO] 2020-06-26 14:41:15,824 [     main.py:  103]:\tepoch 4 | step 70 | loss 1601.681396\n",
      "[INFO] 2020-06-26 14:41:31,381 [     main.py:  114]:\tvalid result: | rmsle 0.33738222738482104 \n",
      "[INFO] 2020-06-26 14:42:05,358 [     main.py:  103]:\tepoch 4 | step 75 | loss 6963.198242\n",
      "[INFO] 2020-06-26 14:42:39,445 [     main.py:  103]:\tepoch 4 | step 80 | loss 44315.679688\n",
      "[INFO] 2020-06-26 14:42:55,124 [     main.py:  114]:\tvalid result: | rmsle 0.4920501171182198 \n",
      "[INFO] 2020-06-26 14:43:29,205 [     main.py:  103]:\tepoch 4 | step 85 | loss 2973.472900\n",
      "[INFO] 2020-06-26 14:44:03,215 [     main.py:  103]:\tepoch 5 | step 90 | loss 2044.652954\n",
      "[INFO] 2020-06-26 14:44:18,849 [     main.py:  114]:\tvalid result: | rmsle 0.34906841902967184 \n",
      "[INFO] 2020-06-26 14:44:52,696 [     main.py:  103]:\tepoch 5 | step 95 | loss 2296.614746\n",
      "[INFO] 2020-06-26 14:45:26,765 [     main.py:  103]:\tepoch 5 | step 100 | loss 32906.230469\n",
      "[INFO] 2020-06-26 14:45:42,225 [     main.py:  114]:\tvalid result: | rmsle 0.3861702873697199 \n",
      "[INFO] 2020-06-26 14:46:16,294 [     main.py:  103]:\tepoch 5 | step 105 | loss 16162.965820\n",
      "[INFO] 2020-06-26 14:46:50,673 [     main.py:  103]:\tepoch 5 | step 110 | loss 2610.524170\n",
      "[INFO] 2020-06-26 14:47:06,688 [     main.py:  114]:\tvalid result: | rmsle 0.36127532195454476 \n",
      "[INFO] 2020-06-26 14:47:41,135 [     main.py:  103]:\tepoch 6 | step 115 | loss 1780.250732\n",
      "[INFO] 2020-06-26 14:48:15,459 [     main.py:  103]:\tepoch 6 | step 120 | loss 10502.518555\n",
      "[INFO] 2020-06-26 14:48:31,190 [     main.py:  114]:\tvalid result: | rmsle 0.3130958394801865 \n",
      "[INFO] 2020-06-26 14:49:05,392 [     main.py:  103]:\tepoch 6 | step 125 | loss 35854.316406\n",
      "[INFO] 2020-06-26 14:49:39,823 [     main.py:  103]:\tepoch 6 | step 130 | loss 2997.706787\n",
      "[INFO] 2020-06-26 14:49:55,792 [     main.py:  114]:\tvalid result: | rmsle 0.34820518475746004 \n",
      "[INFO] 2020-06-26 14:50:30,806 [     main.py:  103]:\tepoch 7 | step 135 | loss 2552.869141\n",
      "[INFO] 2020-06-26 14:51:05,198 [     main.py:  103]:\tepoch 7 | step 140 | loss 3003.099121\n",
      "[INFO] 2020-06-26 14:51:20,733 [     main.py:  114]:\tvalid result: | rmsle 0.3653511384072864 \n",
      "[INFO] 2020-06-26 14:51:54,912 [     main.py:  103]:\tepoch 7 | step 145 | loss 40328.570312\n",
      "[INFO] 2020-06-26 14:52:29,192 [     main.py:  103]:\tepoch 7 | step 150 | loss 3527.086670\n",
      "[INFO] 2020-06-26 14:52:45,366 [     main.py:  114]:\tvalid result: | rmsle 0.4565134441321422 \n",
      "[INFO] 2020-06-26 14:53:19,548 [     main.py:  103]:\tepoch 8 | step 155 | loss 3548.696045\n",
      "[INFO] 2020-06-26 14:53:54,068 [     main.py:  103]:\tepoch 8 | step 160 | loss 1838.059937\n",
      "[INFO] 2020-06-26 14:54:09,859 [     main.py:  114]:\tvalid result: | rmsle 0.3835494900254456 \n",
      "[INFO] 2020-06-26 14:54:43,921 [     main.py:  103]:\tepoch 8 | step 165 | loss 16120.377930\n",
      "[INFO] 2020-06-26 14:55:18,154 [     main.py:  103]:\tepoch 8 | step 170 | loss 26304.166016\n",
      "[INFO] 2020-06-26 14:55:34,055 [     main.py:  114]:\tvalid result: | rmsle 0.47084887527540514 \n",
      "[INFO] 2020-06-26 14:56:08,455 [     main.py:  103]:\tepoch 8 | step 175 | loss 4118.551758\n",
      "[INFO] 2020-06-26 14:56:42,696 [     main.py:  103]:\tepoch 9 | step 180 | loss 3040.967285\n",
      "[INFO] 2020-06-26 14:56:58,365 [     main.py:  114]:\tvalid result: | rmsle 0.37827040902171283 \n",
      "[INFO] 2020-06-26 14:57:32,668 [     main.py:  103]:\tepoch 9 | step 185 | loss 4641.135742\n",
      "[INFO] 2020-06-26 14:58:06,828 [     main.py:  103]:\tepoch 9 | step 190 | loss 36007.191406\n",
      "[INFO] 2020-06-26 14:58:22,481 [     main.py:  114]:\tvalid result: | rmsle 0.3609482683381391 \n",
      "[INFO] 2020-06-26 14:58:56,981 [     main.py:  103]:\tepoch 9 | step 195 | loss 2561.531738\n",
      "[INFO] 2020-06-26 14:59:31,859 [     main.py:  103]:\tepoch 10 | step 200 | loss 4813.833008\n",
      "[INFO] 2020-06-26 14:59:48,230 [     main.py:  114]:\tvalid result: | rmsle 0.3453318222476775 \n",
      "[INFO] 2020-06-26 15:00:23,296 [     main.py:  103]:\tepoch 10 | step 205 | loss 1748.335938\n",
      "[INFO] 2020-06-26 15:00:58,044 [     main.py:  103]:\tepoch 10 | step 210 | loss 27014.339844\n",
      "[INFO] 2020-06-26 15:01:13,904 [     main.py:  114]:\tvalid result: | rmsle 0.3741153056705634 \n",
      "[INFO] 2020-06-26 15:01:48,446 [     main.py:  103]:\tepoch 10 | step 215 | loss 12417.771484\n",
      "[INFO] 2020-06-26 15:02:22,801 [     main.py:  103]:\tepoch 10 | step 220 | loss 4376.041016\n",
      "[INFO] 2020-06-26 15:02:38,670 [     main.py:  114]:\tvalid result: | rmsle 0.394630465021254 \n",
      "[INFO] 2020-06-26 15:02:38,671 [     main.py:  122]:\tbest valid result: 0.31293732133227786\n"
     ]
    }
   ],
   "source": [
    "%run main.py --batch_size 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 交通流量预测\n",
    "## 数据集介绍\n",
    "### 路段属性表\n",
    "每条道路的每个通行方向由多条路段（link）构成，数据集中会提供每条link的唯一标识，长度，宽度，以及道路类型。\n",
    "\n",
    "![file](http://aliyuntianchipublic.cn-hangzhou.oss-pub.aliyun-inc.com/public/files/image/1095279213540/1530604123635_UwFWal5fag.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>link_ID</th>\n",
       "      <th>length</th>\n",
       "      <th>width</th>\n",
       "      <th>link_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4377906289869500514</td>\n",
       "      <td>57</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4377906284594800514</td>\n",
       "      <td>247</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4377906289425800514</td>\n",
       "      <td>194</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4377906284525800514</td>\n",
       "      <td>839</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4377906284422600514</td>\n",
       "      <td>55</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               link_ID  length  width  link_class\n",
       "0  4377906289869500514      57      3           1\n",
       "1  4377906284594800514     247      9           1\n",
       "2  4377906289425800514     194      3           1\n",
       "3  4377906284525800514     839      3           1\n",
       "4  4377906284422600514      55     12           1"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gy_link_info = pd.read_csv('data/data40468/gy_link_info.txt',sep=';')\n",
    "gy_link_info.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "###  link上下游关系表\n",
    "link之间按照车辆允许通行的方向存在上下游关系，数据集中提供每条link的直接上游link和直接下游link。\n",
    "\n",
    "![file](http://aliyuntianchipublic.cn-hangzhou.oss-pub.aliyun-inc.com/public/files/image/1095279213540/1530604200365_YWdoFt080j.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>link_ID</th>\n",
       "      <th>in_links</th>\n",
       "      <th>out_links</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4377906289869500514</td>\n",
       "      <td>4377906285525800514</td>\n",
       "      <td>4377906281969500514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4377906284594800514</td>\n",
       "      <td>4377906284514600514</td>\n",
       "      <td>4377906285594800514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4377906289425800514</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4377906284653600514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4377906284525800514</td>\n",
       "      <td>4377906281234600514</td>\n",
       "      <td>4377906280334600514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4377906284422600514</td>\n",
       "      <td>3377906289434510514#4377906287959500514</td>\n",
       "      <td>4377906283422600514</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               link_ID                                 in_links  \\\n",
       "0  4377906289869500514                      4377906285525800514   \n",
       "1  4377906284594800514                      4377906284514600514   \n",
       "2  4377906289425800514                                      NaN   \n",
       "3  4377906284525800514                      4377906281234600514   \n",
       "4  4377906284422600514  3377906289434510514#4377906287959500514   \n",
       "\n",
       "             out_links  \n",
       "0  4377906281969500514  \n",
       "1  4377906285594800514  \n",
       "2  4377906284653600514  \n",
       "3  4377906280334600514  \n",
       "4  4377906283422600514  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gy_link_top = pd.read_csv('data/data40468/gy_link_top.txt',sep=';')\n",
    "gy_link_top.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### link历史通行时间表\n",
    "数据集中记录了历史每天不同时间段内（2min为一个时间段）每条link上的平均旅行时间，每个时间段的平均旅行时间是基于在该时间段内进入link的车辆在该link上的旅行时间产出。\n",
    "\n",
    "![file](http://aliyuntianchipublic.cn-hangzhou.oss-pub.aliyun-inc.com/public/files/image/1095279213540/1530604254989_39m8JehJpp.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  data/data40468/travel_time_1.zip\n",
      "replace work/dataset/gy_link_travel_time_part1.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: ^C\n",
      "Archive:  data/data40468/travel_time_2.zip\n",
      "replace work/dataset/gy_link_travel_time_part2.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: "
     ]
    }
   ],
   "source": [
    "!unzip data/data40468/travel_time_1.zip -d work/dataset/\n",
    "!unzip data/data40468/travel_time_2.zip -d work/dataset/\n",
    "!unzip data/data40468/travel_time_3.zip -d work/dataset/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3018: DtypeWarning: Columns (0,3) have mixed types. Specify dtype option on import or set low_memory=False.\n",
      "  interactivity=interactivity, compiler=compiler, result=result)\n"
     ]
    }
   ],
   "source": [
    "df1 = pd.read_csv('work/dataset/gy_link_travel_time_part1.txt',sep=';')\n",
    "df2 = pd.read_csv('work/dataset/gy_link_travel_time_part2.txt',sep=';', names=['link_ID','date','time_interval','travel_time'])\n",
    "df2.drop(0,inplace=True)\n",
    "df3 = pd.read_csv('work/dataset/gy_link_travel_time_part3.txt',sep=';')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>link_ID</th>\n",
       "      <th>date</th>\n",
       "      <th>time_interval</th>\n",
       "      <th>travel_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10632039</th>\n",
       "      <td>4377906287663800514</td>\n",
       "      <td>2017-07-31</td>\n",
       "      <td>[2017-07-31 14:16:00,2017-07-31 14:18:00)</td>\n",
       "      <td>76.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10632040</th>\n",
       "      <td>4377906288663800514</td>\n",
       "      <td>2017-07-31</td>\n",
       "      <td>[2017-07-31 07:08:00,2017-07-31 07:10:00)</td>\n",
       "      <td>3.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10632041</th>\n",
       "      <td>4377906288663800514</td>\n",
       "      <td>2017-07-31</td>\n",
       "      <td>[2017-07-31 17:12:00,2017-07-31 17:14:00)</td>\n",
       "      <td>5.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10632042</th>\n",
       "      <td>4377906288663800514</td>\n",
       "      <td>2017-07-31</td>\n",
       "      <td>[2017-07-31 17:38:00,2017-07-31 17:40:00)</td>\n",
       "      <td>36.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10632043</th>\n",
       "      <td>3377906289044510514</td>\n",
       "      <td>2017-07-31</td>\n",
       "      <td>[2017-07-31 17:08:00,2017-07-31 17:10:00)</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      link_ID        date  \\\n",
       "10632039  4377906287663800514  2017-07-31   \n",
       "10632040  4377906288663800514  2017-07-31   \n",
       "10632041  4377906288663800514  2017-07-31   \n",
       "10632042  4377906288663800514  2017-07-31   \n",
       "10632043  3377906289044510514  2017-07-31   \n",
       "\n",
       "                                      time_interval travel_time  \n",
       "10632039  [2017-07-31 14:16:00,2017-07-31 14:18:00)        76.9  \n",
       "10632040  [2017-07-31 07:08:00,2017-07-31 07:10:00)         3.4  \n",
       "10632041  [2017-07-31 17:12:00,2017-07-31 17:14:00)         5.1  \n",
       "10632042  [2017-07-31 17:38:00,2017-07-31 17:40:00)        36.6  \n",
       "10632043  [2017-07-31 17:08:00,2017-07-31 17:10:00)          11  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gy_link_travel_time = pd.concat([df1, df2, df3], axis=0)\n",
    "gy_link_travel_time.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "del df1\n",
    "del df2\n",
    "del df3\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gy_link_travel_time['time_interval'] = gy_link_travel_time['time_interval'].str(1:20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "264"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(gy_link_travel_time['link_ID'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tmp = gy_link_travel_time[['time_interval', 'link_ID', 'travel_time']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tmp.to_csv('work/dataset/gy_link_travel_time.csv',index=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tmp = pd.read_csv('work/dataset/gy_link_travel_time.csv',parse_dates=['travel_time'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time_interval</th>\n",
       "      <th>link_ID</th>\n",
       "      <th>travel_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016-05-21 23:20:00</td>\n",
       "      <td>9377906285566510514</td>\n",
       "      <td>17.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016-05-21 18:46:00</td>\n",
       "      <td>3377906288228510514</td>\n",
       "      <td>3.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016-05-21 07:06:00</td>\n",
       "      <td>3377906284395510514</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2016-05-21 14:34:00</td>\n",
       "      <td>4377906284959500514</td>\n",
       "      <td>3.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2016-05-21 05:04:00</td>\n",
       "      <td>9377906282776510514</td>\n",
       "      <td>1.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         time_interval              link_ID travel_time\n",
       "0  2016-05-21 23:20:00  9377906285566510514        17.6\n",
       "1  2016-05-21 18:46:00  3377906288228510514         3.5\n",
       "2  2016-05-21 07:06:00  3377906284395510514        10.0\n",
       "3  2016-05-21 14:34:00  4377906284959500514         3.5\n",
       "4  2016-05-21 05:04:00  9377906282776510514         1.5"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tmp['travel_time'] = tmp['travel_time'].astype(np.float16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tmp = tmp.groupby(['time_interval','link_ID']).agg({'travel_time': ['mean']})\n",
    "tmp.columns = ['travel_time']\n",
    "tmp.reset_index(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time_interval</th>\n",
       "      <th>link_ID</th>\n",
       "      <th>travel_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016-03-01 00:00:00</td>\n",
       "      <td>3377906280028510514</td>\n",
       "      <td>4.601562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016-03-01 00:00:00</td>\n",
       "      <td>3377906280395510514</td>\n",
       "      <td>22.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016-03-01 00:00:00</td>\n",
       "      <td>3377906282328510514</td>\n",
       "      <td>20.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2016-03-01 00:00:00</td>\n",
       "      <td>3377906283328510514</td>\n",
       "      <td>6.601562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2016-03-01 00:00:00</td>\n",
       "      <td>3377906284028510514</td>\n",
       "      <td>19.593750</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         time_interval              link_ID  travel_time\n",
       "0  2016-03-01 00:00:00  3377906280028510514     4.601562\n",
       "1  2016-03-01 00:00:00  3377906280395510514    22.500000\n",
       "2  2016-03-01 00:00:00  3377906282328510514    20.000000\n",
       "3  2016-03-01 00:00:00  3377906283328510514     6.601562\n",
       "4  2016-03-01 00:00:00  3377906284028510514    19.593750"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 生成完整时间段序列，解决空值问题"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['2016-03-01 00:00:00', '2016-03-01 00:02:00',\n",
       "       '2016-03-01 00:04:00', ..., '2017-07-31 17:54:00',\n",
       "       '2017-07-31 17:56:00', '2017-07-31 17:58:00'], dtype=object)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp['time_interval'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "183624"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tmp['time_interval'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 生成完整的时间段序列\n",
    "ts = pd.Series(np.zeros(len(tmp['time_interval'].unique())), index=tmp['time_interval'].unique())\n",
    "ts.to_csv(\"data/ts.csv\", header=None)\n",
    "ts = pd.read_csv(\"data/ts.csv\", names=['time_interval', 'value'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# tmp = pd.merge(ts,tmp,how='left')\n",
    "# tmp.drop(['value'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 依样画葫芦\n",
    "### 通行时间处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def car_process(tmp, ts, output_path):\n",
    "    res = []\n",
    "    link_name_list = []\n",
    "        # order = sorted(range(region_nums[i]), key=lambda x:str(x))\n",
    "    order = sorted(tmp['link_ID'].unique())\n",
    "    for i, idx in enumerate(order):\n",
    "        target_link = idx #str(idx)\n",
    "        # pdb.set_trace()\n",
    "        # 路段平均车速\n",
    "        df = tmp[tmp['link_ID'] == target_link].reset_index(drop=True)\n",
    "        df = pd.merge(ts,df,how='left')\n",
    "        df.drop(['value'], axis=1, inplace=True)\n",
    "        if i == 0:\n",
    "            # 第一个路段要把时间给进去\n",
    "            df = df[['time_interval', 'travel_time']]\n",
    "        else:\n",
    "            df = df[['travel_time']]\n",
    "\n",
    "        df = df.rename(columns={'travel_time': '%d' % (idx)})\n",
    "        link_name_list.append(\"%d\" % (idx))\n",
    "\n",
    "        res.append(df)\n",
    "    df = pd.concat(res, axis=1)\n",
    "    # 最终形成路段ID形式的平均车速大宽表\n",
    "    file_to_save = os.path.join(output_path, \"travel_time.csv\")\n",
    "    print(\"saving result to %s\" % file_to_save)\n",
    "    df.to_csv(file_to_save, index=False)\n",
    "\n",
    "    link_name_file = os.path.join(output_path, \"link_name_list.txt\")\n",
    "    with open(link_name_file, 'w') as f:\n",
    "        names = ' '.join(link_name_list)\n",
    "        # print(names)\n",
    "        f.write(names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving result to work/dataset/travel_time.csv\n"
     ]
    }
   ],
   "source": [
    "car_process(tmp,ts,'work/dataset')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "travel_time = pd.read_csv('work/dataset/travel_time.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time_interval</th>\n",
       "      <th>3377906280028510514</th>\n",
       "      <th>3377906280395510514</th>\n",
       "      <th>3377906281518510514</th>\n",
       "      <th>3377906281774510514</th>\n",
       "      <th>3377906282328510514</th>\n",
       "      <th>3377906282418510514</th>\n",
       "      <th>3377906283328510514</th>\n",
       "      <th>3377906284028510514</th>\n",
       "      <th>3377906284395510514</th>\n",
       "      <th>...</th>\n",
       "      <th>9377906282776510514</th>\n",
       "      <th>9377906283125510514</th>\n",
       "      <th>9377906283776510514</th>\n",
       "      <th>9377906284555510514</th>\n",
       "      <th>9377906285566510514</th>\n",
       "      <th>9377906285615510514</th>\n",
       "      <th>9377906286566510514</th>\n",
       "      <th>9377906286615510514</th>\n",
       "      <th>9377906288175510514</th>\n",
       "      <th>9377906289175510514</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>183619</th>\n",
       "      <td>2017-07-31 17:50:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>18.6</td>\n",
       "      <td>4.5</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>21.4</td>\n",
       "      <td>3.4</td>\n",
       "      <td>...</td>\n",
       "      <td>1.6</td>\n",
       "      <td>18.7</td>\n",
       "      <td>15.3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>66.7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183620</th>\n",
       "      <td>2017-07-31 17:52:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>21.3</td>\n",
       "      <td>6.1</td>\n",
       "      <td>11.6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.7</td>\n",
       "      <td>3.6</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>19.2</td>\n",
       "      <td>12.6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16.9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>66.9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12.0</td>\n",
       "      <td>3.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183621</th>\n",
       "      <td>2017-07-31 17:54:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>21.4</td>\n",
       "      <td>8.5</td>\n",
       "      <td>9.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>28.9</td>\n",
       "      <td>3.9</td>\n",
       "      <td>...</td>\n",
       "      <td>2.9</td>\n",
       "      <td>22.4</td>\n",
       "      <td>15.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>15.7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>55.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.3</td>\n",
       "      <td>3.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183622</th>\n",
       "      <td>2017-07-31 17:56:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>21.5</td>\n",
       "      <td>7.7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>24.6</td>\n",
       "      <td>3.5</td>\n",
       "      <td>...</td>\n",
       "      <td>2.9</td>\n",
       "      <td>19.5</td>\n",
       "      <td>13.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>15.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>47.9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.1</td>\n",
       "      <td>3.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183623</th>\n",
       "      <td>2017-07-31 17:58:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>24.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>27.3</td>\n",
       "      <td>3.4</td>\n",
       "      <td>...</td>\n",
       "      <td>2.5</td>\n",
       "      <td>24.0</td>\n",
       "      <td>15.3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>42.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.1</td>\n",
       "      <td>3.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 133 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              time_interval  3377906280028510514  3377906280395510514  \\\n",
       "183619  2017-07-31 17:50:00                  NaN                 18.6   \n",
       "183620  2017-07-31 17:52:00                  NaN                 21.3   \n",
       "183621  2017-07-31 17:54:00                  NaN                 21.4   \n",
       "183622  2017-07-31 17:56:00                  NaN                 21.5   \n",
       "183623  2017-07-31 17:58:00                  NaN                 24.0   \n",
       "\n",
       "        3377906281518510514  3377906281774510514  3377906282328510514  \\\n",
       "183619                  4.5                 10.0                  NaN   \n",
       "183620                  6.1                 11.6                  NaN   \n",
       "183621                  8.5                  9.2                  NaN   \n",
       "183622                  7.7                  NaN                  NaN   \n",
       "183623                  4.5                  NaN                  NaN   \n",
       "\n",
       "        3377906282418510514  3377906283328510514  3377906284028510514  \\\n",
       "183619                  3.9                  NaN                 21.4   \n",
       "183620                  3.7                  NaN                 26.7   \n",
       "183621                  4.8                  NaN                 28.9   \n",
       "183622                  5.1                  NaN                 24.6   \n",
       "183623                  5.2                  NaN                 27.3   \n",
       "\n",
       "        3377906284395510514         ...           9377906282776510514  \\\n",
       "183619                  3.4         ...                           1.6   \n",
       "183620                  3.6         ...                           2.0   \n",
       "183621                  3.9         ...                           2.9   \n",
       "183622                  3.5         ...                           2.9   \n",
       "183623                  3.4         ...                           2.5   \n",
       "\n",
       "        9377906283125510514  9377906283776510514  9377906284555510514  \\\n",
       "183619                 18.7                 15.3                  NaN   \n",
       "183620                 19.2                 12.6                  NaN   \n",
       "183621                 22.4                 15.0                  NaN   \n",
       "183622                 19.5                 13.4                  NaN   \n",
       "183623                 24.0                 15.3                  NaN   \n",
       "\n",
       "        9377906285566510514  9377906285615510514  9377906286566510514  \\\n",
       "183619                 16.0                  NaN                 66.7   \n",
       "183620                 16.9                  NaN                 66.9   \n",
       "183621                 15.7                  NaN                 55.0   \n",
       "183622                 15.5                  NaN                 47.9   \n",
       "183623                 16.1                  NaN                 42.1   \n",
       "\n",
       "        9377906286615510514  9377906288175510514  9377906289175510514  \n",
       "183619                  NaN                 12.0                  2.9  \n",
       "183620                  NaN                 12.0                  3.1  \n",
       "183621                  NaN                 10.3                  3.7  \n",
       "183622                  NaN                 10.1                  3.7  \n",
       "183623                  NaN                 11.1                  3.9  \n",
       "\n",
       "[5 rows x 133 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "travel_time.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "travel_time['time_interval'].dropna(axis=0, how='any', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 迁移车流量处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def process_car_migration(data_path, city_name):\n",
    "    filename = os.path.join(data_path, \"city_%s\" % city_name, \"migration.csv\")\n",
    "    migration = pd.read_csv(filename, \n",
    "                            sep=',', \n",
    "                            header=None,\n",
    "                            names=['date', 's_city', 'e_city', city_name])\n",
    "\n",
    "    # only use moving in \"city\" data, ignore moving out data\n",
    "    df = migration[migration.e_city == city_name]\n",
    "    df = df[[\"date\", city_name]]\n",
    "\n",
    "    # calculate total move in data of \"city\"\n",
    "    df = df.groupby('date')[city_name].sum().reset_index()\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# STGCN：时空图卷积网络\n",
    "> 相关论文：[Spatio-Temporal Graph Convolutional Network \\(STGCN\\)](https://arxiv.org/pdf/1709.04875.pdf) \n",
    "在PGL中，提供了使用STGCN进行新冠疫情感染者趋势预测的示例。\n",
    "## 数据集\n",
    "需要将数据集按照下面的格式进行整理：\n",
    "- input.csv: 历史迁移数据 with shape of [num\\_time\\_steps * num\\_cities].\n",
    "\n",
    "-  output.csv: 新增确诊数据 with shape of [num\\_time\\_steps * num\\_cities].\n",
    "\n",
    "- W.csv: 权重邻接矩阵 with shape of [num\\_cities * num\\_cities].\n",
    "\n",
    "- city.csv: 城市列表.\n",
    "\n",
    "### 开始训练\n",
    "\n",
    "使用GPU训练示例\n",
    "```\n",
    "python main.py --use_cuda --input_file dataset/input_csv --label_file dataset/output.csv --adj_mat_file dataset/W.csv --city_file dataset/city.csv \n",
    "```\n",
    "\n",
    "## 超参数\n",
    "\n",
    "- n\\_route: Number of city.\n",
    "- n\\_his: \"n\\_his\" time steps of previous observations of historical immigration records.\n",
    "- n\\_pred: Next \"n\\_pred\" time steps of New confirmed patients records.\n",
    "- Ks: Number of GCN layers.\n",
    "- Kt: Kernel size of temporal convolution.\n",
    "- use\\_cuda: Use gpu if assign use\\_cuda. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# STGCN: Spatio-Temporal Graph Convolutional Network\n",
    "\n",
    "[Spatio-Temporal Graph Convolutional Network \\(STGCN\\)](https://arxiv.org/pdf/1709.04875.pdf) is a novel deep learning framework to tackle time series prediction problem. Based on PGL, we reproduce STGCN algorithms to predict new confirmed patients in some cities with the historical immigration records.\n",
    "\n",
    "### Datasets\n",
    "\n",
    "You can make your customized dataset by the following format:\n",
    "\n",
    "* input.csv: Historical immigration records with shape of [num\\_time\\_steps * num\\_cities].\n",
    "\n",
    "* output.csv: New confirmed patients records with shape of [num\\_time\\_steps * num\\_cities].\n",
    "\n",
    "* W.csv: Weighted Adjacency Matrix with shape of [num\\_cities * num\\_cities].\n",
    "\n",
    "* city.csv: Each line is a number and the corresponding city name.\n",
    "\n",
    "### Dependencies\n",
    "\n",
    "- paddlepaddle 1.6\n",
    "- pgl 1.0.0\n",
    "\n",
    "### How to run\n",
    "\n",
    "For examples, use gpu to train STGCN on your dataset.\n",
    "```\n",
    "python main.py --use_cuda --input_file dataset/input_csv --label_file dataset/output.csv --adj_mat_file dataset/W.csv --city_file dataset/city.csv \n",
    "```\n",
    "\n",
    "#### Hyperparameters\n",
    "\n",
    "- n\\_route: Number of city.\n",
    "- n\\_his: \"n\\_his\" time steps of previous observations of historical immigration records.\n",
    "- n\\_pred: Next \"n\\_pred\" time steps of New confirmed patients records.\n",
    "- Ks: Number of GCN layers.\n",
    "- Kt: Kernel size of temporal convolution.\n",
    "- use\\_cuda: Use gpu if assign use\\_cuda. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "请点击[此处](https://ai.baidu.com/docs#/AIStudio_Project_Notebook/a38e5576)查看本环境基本用法.  <br>\n",
    "Please click [here ](https://ai.baidu.com/docs#/AIStudio_Project_Notebook/a38e5576) for more detailed instructions. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PaddlePaddle 1.8.0 (Python 3.5)",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
